Congratulations to CSD faculty Aditi Raghunathan and her research collaborators on receiving an ICML Outstanding Paper award for Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction (icml.cc/virtual/2025...). Paper: arxiv.org/abs/2504.15266

Reading the dedications of a PhD thesis is often a cure for a bad day. There’s so much affection in them

As NeurIPS review deadline is around the corner, please remember that you cannot use any non-local LLM like chatgpt/gemini for understanding the paper and drafting/revising your review as that breaks the confidentiality agreement. NeurIPS 2025 Official LLM Policy: neurips.cc/Conferences/...

I really enjoyed "When We Cease to Understand the World", although it's more fiction than history of science

“Science in history” by Bernal is my first recommendation. The work of Ian Hacking is a good recommendation for Probability

How do task dynamics impact learning in networks with internal dynamics? Excited to share our ICML Oral paper on learning dynamics in linear RNNs! with @clementinedomine.bsky.social @mpshanahan.bsky.social and Pedro Mediano openreview.net/forum?id=KGO...

When we are doing science, we are unknowingly executing our mythology, taken from movies and friends and textbooks, of what science is. History of science helps us ground that myth in reality

I finally wrote a full-fledged blog about this: reading the history of science is an **amazing** yet under-recognized way to develop (emotional) maturity as a researcher. If you have thoughts/recommendations, please share! vaishnavh.github.io/2025/04/29/h...

A string may get 17 times less probability if tokenised as two symbols (e.g., ⟨he, llo⟩) than as one (e.g., ⟨hello⟩)—by an LM trained from scratch in each situation! Our new ACL paper proposes an observational method to estimate this causal effect! Longer thread soon!

How does in-context learning emerge in attention models during gradient descent training? Sharing our new Spotlight paper @icmlconf.bsky.social: Training Dynamics of In-Context Learning in Linear Attention arxiv.org/abs/2501.16265 Led by Yedi Zhang with @aaditya6284.bsky.social and Peter Latham

This paper is quite nice. It mixes some useful toy models of creativity with insights about how to induce more creativity in LLMs that are better than greedy sampling

📢 New #paper on creativity & multi-token prediction! We design minimal open-ended tasks to argue: → LLMs are limited in creativity as they learn to predict the next token → creativity can be improved via multi-token learning & injecting noise ("seed-conditioning" 🌱) 1/ #MLSky #AI #arxiv 🧵👇🏽

This isn't fake news. One of the craziest AI research papers I've been on in a while. Weird ablations on RLVR shows that the Qwen 2.5 models can learn with literally random rewards, likely due to some funkiness in mid-training and the GRPO setup.

can someone reconcile these two contradictory findings?! two papers find entropy *minimization*/confidence maximization helps performance, and the RL-on-one-sample finds entropy maximization/increasing exploration alone helps performance?!

🚨 New Paper! A lot happens in the world every day—how can we update LLMs with belief-changing news? We introduce a new dataset "New News" and systematically study knowledge integration via System-2 Fine-Tuning (Sys2-FT). 1/n

The contextual shadowing effect in this paper is interesting & reminds me of the infamous "cow-in-a-beach" spurious correlation examples. The model needs to learn a core feature (here. the store-in-memory feature) but it instead relies on a simpler "spurious" feature (the in-context feature).

My #neurips bidding pool is REALLY bad and I see that I'm not alone. Was the pool restricted to 100 papers to break collusion? if so, seems like everyone else is going to pay for this

looks like a fun workshop!

Excited that our paper "safety alignment should be made more than just a few tokens deep" was recognized as an #ICLR2025 Outstanding Paper! We identified a common root cause to many safety vulnerabilities and pointed out some paths forward to address it!

At the moment, we just show you posts by people you follow. Following someone will make their paper posts appear! In the future, we'll expand this (slightly) to show you other paper posts that may be of interest

Yeah this is actual unironic advice for any 1st years: you seriously need to go get beers or otherwise hangout socially with more senior grad students (no one else will really know) and hopefully get ensconced enough that someone will tell you “hey just fyi Dr. GoodPubs is low-key a total sociopath”

@vaishnavh.bsky.social and crew do it again: - a benchmark for open-ended creativity - a demonstration of challenges of next-token prediction - a technique to improve transformer randomness through inputs not sampling arxiv.org/abs/2504.15266

@icmlconf.bsky.social (assuming this is the legit ICML account), the Canadian visa application requires providing the inviting contact to have a Canadian address. But the one in the visa letter is a US address. Could you look into this?

If you are at #AISTATS2025 and are interested in concept erasure, talk to @somnathbrc.bsky.social at Poster Session 1 on Saturday May 3.

Responsible reviewing initiatives for NeurIPS 2025 - read more about changes to reviewing that that will safeguard reviewing quality and timeline in our blog post below: blog.neurips.cc/2025/05/02/r...

Self-improving Transformers can overcome easy-to-hard and length generalization challenges. Paper on arxiv coming on Monday. Link to a talk I gave on this below 👇 Super excited about this work! Talk : youtube.com/watch?v=szhE... slides: tinyurl.com/SelfImprovem...

I wrote a post on how to connect with people (i.e., make friends) at CS conferences. These events can be intimidating so here's some suggestions on how to navigate them I'm late for #ICLR2025 #NAACL2025, but in time for #AISTATS2025 #ICML2025! 1/3 kamathematics.wordpress.com/2025/05/01/t...

Thrilled to announce that Joshua’s paper won one of three Outstanding Paper awards at ICLR. Come to the poster on Friday afternoon (#376 in Hall 3) or the talk on Saturday (4:30 in Hall 1), and while you’re at it snag him for a postdoc!

Been waiting for a work making this clear for a while: zero-shot, LLMs do not hedge and do not ask questions to solve under-specified problems arxiv.org/abs/2503.22674

What tools do people use to manage AC/meta-reviewing responsibilities (like consolidating key points, flagging which papers/reviews need attention, etc., etc.,)?

o3 can't multiply beyond a few digits... But I think multiplication, addition, maze solving and easy-to-hard generalization is actually solvable on standard transformers... with recursive self-improvement Below is the acc of a tiny model teaching itself how to add and multiply

Training with more data = better LLMs, right? 🚨 False! Scaling language models by adding more pre-training data can decrease your performance after post-training! Introducing "catastrophic overtraining." 🥁🧵👇 arxiv.org/abs/2503.19206 1/10

Come speak at our recurring "research jam" — the "lab meeting" equivalent of an independent and distributed research community this Wednesday, if you are working on a fun project and seeking feedback, collaboration, and/or compute resources! mlcollective.org/events/resea...

Excited to share our work with friends from MIT/Google on Learned Asynchronous Decoding! LLM responses often contain chunks of tokens that are semantically independent. What if we can train LLMs to identify such chunks and decode them in parallel, thereby speeding up inference? 1/N

My Simons talk on all the next-token & multi-token prediction stuff I've been yapping about is up, if anyone wants to watch! There're better versions of the talk that Gregor co-presented with me elsewhere, but sadly not recorded. He is an amazing presenter! www.youtube.com/watch?v=9V0b...

Task vectors are akin to punchcards: you feed them to your LLM and it implements specific tasks, without in-context demonstrations. Liu's new paper examines at what scale, where in the network and when during training do they emerge, and how to encourage their emergence. arxiv.org/pdf/2501.09240

📢 My team at Meta (including Yaron Lipman and Ricky Chen) is hiring a postdoctoral researcher to help us build the next generation of flow, transport, and diffusion models! Please apply here and message me: www.metacareers.com/jobs/1459691...

Interesting paper showing how LLMs change their representational geometry in-context to match a task structure: arxiv.org/abs/2501.00070 Here, the model’s latent representations show a grid structure matching the task. #MLSKy #NeuroAI

I've been thinking about in-context learning for nearly 3 years. While there is still plenty I don't fully understand, five papers have--to a very large extent--shaped my perspective on it, and I believe everyone should read them.

The single most insightful paper that I read after the initial wave of ICL papers was one that further shifted my understanding on "the early ascent phenomenon" by Kangwook Lee's group x.com/Kangwook_Lee...

Very interesting paper by Ananda Theertha Suresh et al. For categorical/Gaussian distributions, they derive the rate at which a sample is forgotten to be 1/k after k rounds of recursive training (hence 𝐦𝐨𝐝𝐞𝐥 𝐜𝐨𝐥𝐥𝐚𝐩𝐬𝐞 happens more slowly than intuitively expected)

does next token prediction (NTP) affirm or go against the bitter lesson? funnily, one can argue both ways.

Transformer LMs get pretty far by acting like ngram models, so why do they learn syntax? A new paper by sunnytqin.bsky.social, me, and @dmelis.bsky.social illuminates grammar learning in a whirlwind tour of generalization, grokking, training dynamics, memorization, and random variation. #mlsky #nlp

1/2 Just a reminder about Google Research Scholar Program, providing up to $60K unrestricted gifts to recognize early-career professors and support world-class research at institutions around the world. This year, we are particularly interested in the following research areas...

Transformers can navigate mazes with multi-step prediction (Nolte, Kitouni, et al, arXiv) tldr- MLM-U enc-dec transformers w a masking objective seem to crush dec transformers trained with next token prediction objectives on 10x10 to 30x30 maze solving (with small models) arxiv.org/abs/2412.05117

Introducing the @tmlr-pub.bsky.social account, that is now posting the latest TMLR published papers! Check it out!

Can language models transcend the limitations of training data? We train LMs on a formal grammar, then prompt them OUTSIDE of this grammar. We find that LMs often extrapolate logical rules and apply them OOD, too. Proof of a useful inductive bias. Check it out at NeurIPS: nips.cc/virtual/2024...

📣I'm hiring PhD interns for combined theory+empirical projects in: exploration in post-training, multi-task learning in autoregressive models, distillation, reasoning beyond CoT. Apply on the link below. If you're at #NeurIPS2024, message me to chat. jobs.careers.microsoft.com/global/en/jo...

hi everyone!! let's try this optimal transport again 🙃

this worked! thanks to nominative determinism, Brandon helped me put the bsky brand on my website