It took me weeks, but finally it's there: an overlong blogpost on synthetic pretraining. vintagedata.org/blog/posts/s...

this lecture series is really useful for people who took the roundabout route to programming: missing.csail.mit.edu

Unit tests in research code feel like something that's slowing you down until they wind up saving you six months of suffering

i read through ~5 pages of the Olmo 3 tech report.. whoah this is the best and most detailed summary of the current state of SOTA LLM training nanochat is good for understanding LLM training, this tech report catches you up to SOTA methods

Announcing Olmo 3, a leading fully open LM suite built for reasoning, chat, & tool use, and an open model flow—not just the final weights, but the entire training journey. Best fully open 32B reasoning model & best 32B base model. 🧵

Secure your NLDL 2026 registration before the fees increase on December 1st! 🤖 ❄️ Registration is open until January 1st 2026, but we recommend registering early to avoid expensive hotel prices More info in the comments 👇

Today I learnt that OpenSSL really does not like it if you try to pass in an X.509 certificate which only consists of the word "Blah"

Playing around with the PleIAs "smallest viable model" Monad, and realizing that with 4-bit quantization (storing 56 M parameters in ~27 MB) and a SuperDisk drive (to use the FD32MB format), you could turn it into a chat model that fits on a standard 3.5 inch diskette

We are excited to have Mihaela van der Schaar and Anders Boyd as Winter School speakers at the Northern Lights Deep Learning Conference 2026! Read more about van der Schaar's and Boyd's and other Winter School tutorials in the comments 👇

Having used the Mac for a bit, I realize most of my time is spent in the same software I was using on Windows (Obsidian, Zotero, Marimo), but Homebrew is quite nice, Ghostty is a good terminal, and UTM is a good QEMU/virtualization frontend for the Windows things I do need

> RigAnything: Template-Free Autoregressive Rigging for Diverse 3D Assets Paper: arxiv.org/abs/2502.09615 Web: www.liuisabella.com/RigAnything/ Code: github.com/Isabella98Li... Model: huggingface.co/Isabellaliu/...

Neural audio codecs: how to get audio into LLMs The plan: sandwich a language model in an audio encoder/decoder pair (=neural audio codec), allowing it to predict audio continuations. kyutai.org/next/codec-e...

Deepseek's DeepSeek-OCR Model: huggingface.co/deepseek-ai/... Paper: github.com/deepseek-ai/... Repo: github.com/deepseek-ai/...

Is 32B-4bit equal to 16B-8bit? Depends on the task * math: precision matters * knowledge: effective param count is more important * 4B-8bit threshold — for bigger prefer quant, smaller prefer more params * parallel TTC only works above 4B-8bit arxiv.org/abs/2510.10964

GPU computing before CUDA was *weird*.
 Memory primitives were graphics shaped, not computer science shaped.
 Want to do math on an array? Store it as an RGBA texture. 
Fragment Shader for processing. *Paint* the result in a big rectangle.

nanochat by Andrej Karpathy is neat - 8,000 lines of code (mostly Python, a tiny bit of Rust) that can train an LLM on $100 of rented cloud compute which can then be served with a web chat UI on a much smaller machine simonwillison.net/2025/Oct/13/...

Only 7 days left to submit your abstract for the Northern Lights Deep Learning Conference 2026! 🤖 ❄️ 📅 Abstract submission deadline: October 17th 2025 More information about submission guidelines on nldl.org

Defying Transformers: Searching for "Fixed Points" of Pretrained LLMs by Jiacheng Liu He wondered what CAN'T be transformed by Transformers? So, he wrote a fun blog post on finding "fixed points" of your LLMs. If you prompt it with a fixed point token,

new blog post! why do LLMs freak out over the seahorse emoji? i put llama-3.3-70b through its paces with the logit lens to find out, and explain what the logit lens (everyone's favorite underrated interpretability tool) is in the process. link in reply!

How Diffusion Models Memorize Juyeop Kim, Songkuk Kim, Jong-Seok Lee tl;dr: classifier-free-guidance is to blame arxiv.org/abs/2509.25705

Very happy this paper got accepted to NeurIPS 2025 as a Spotlight! 😁 Main takeaway: In mechanistic interpretability, we need assumptions about how DNNs encode concepts in their representations (eg, the linear representation hypothesis). Without them, we can claim any DNN implements any algorithm!

really neat clear explainer for the new on “centralizing flows” to theoretically model learning dynamics

Half-serious question: Could one tackle every "language models produce X because Y is in the training data" hypothesis in one go by training a large retrieval transformer, and doing side-by-side evaluations with/without a filter for Y on the retriever? #MLSky

And new paper out: Pleias 1.0: the First Family of Language Models Trained on Fully Open Data How we train an open everything model on a new pretraining environment with releasable data (Common Corpus) with an open source framework (Nanotron from HuggingFace). www.sciencedirect.com/science/arti...

Stepping outside my lane for a bit, NPM really needs trusted publishing/provenance, but I think a per-package "will use secure-context-only features" flag would go a long way too, there are few good reasons for a package to suddenly _start_ using eval(), Fetch, or executing arbitrary commands

As someone interested in cryptography and privacy-preserving measures, it's cool to see differential privacy being applied to ML practically like this - this seems to basically stop memorization of passages from the training set, even allowing for approximate matches (up to 10% edit distance)

We've just released an amazing Embedding model: EmbeddingGemma, the new best-in-class open embedding model! 🚀 🏆 Top multilingual model on MTEB (<500M) 💾 Runs on <200MB RAM ⚙️ Customizable output for on-device use 🧩 Integrated with your favorite tools developers.googleblog.com/en/introduci...

EPFL, ETH Zurich & CSCS just released Apertus, Switzerland’s first fully open-source large language model. Trained on 15T tokens in 1,000+ languages, it’s built for transparency, responsibility & the public good. Read more: actu.epfl.ch/news/apertus...

As someone who hasn't used MacOS in a minute (since... Sierra?), which MacOS utilities do people on #MLSky recommend? (Homebrew is a given, but not sure which terminal emulators people prefer now, for instance)

Limits of vector search a new GDM paper shows that embeddings can’t represent combinations of concepts well e.g. Dave likes blue trucks AND Ford trucks even k=2 sub-predicates make SOTA embedding models fall apart www.alphaxiv.org/pdf/2508.21038

NLDL 2026 paper submission deadline extended 🤖 ❄️ Due to a large amount of requests, we wish to inform you that the paper submission deadline is extended until September 14th, 23:59 CEST More info about submissions at nldl.org

Interesting argument that the architectural blocks of language models are almost always surjective, implying that any given output text has at least one corresponding input which can generate it (They do note that *finding* that input isn't necessarily easy, and assume you control input embeddings)

Wonderful project which packages ELIZA and other retro chatbots in an OpenAI-compatible API, because we all need a code assistant that can ask "how does the NullReferenceException make you feel?"

Nvidia have released a new set of hybrid Mamba-Transformer language models, but interestingly they've also published the majority of the data used to pretrain them: research.nvidia.com/labs/adlr/NV...

Fun breakdown of the process of implementing a toy TPU in Verilog, starting "from scratch" www.tinytpu.com

Interesting from DeepMind: Live Music Models arxiv.org/pdf/2508.04651 - low-latency music generation models which run at real time and respond to user input, letting you perform style transfer on your own music on the fly

Shower thought: if you only had pen and paper (no calculator), what kind of neural networks could you feasibly still compute predictions with? I think one could make something like the earliest LeNets work, with ReLU activation and some form of weight quantization to simplify things

Today's small but neat trick: setting up a simple web server in Powershell to serve a Marimo notebook to localhost, giving you a light Python + Numpy + Matplotlib setup on any Windows box without any installation or network requirements