Absolute joy to present with Sarah Immel the paper we wrote with @neelrajani.bsky.social - Stepping Into the Black Box: Opening Up LLMs for Public Exploration Through Discursive Design at @acm-dis.bsky.social in Singapore. And we won an Honourable Mention!

NLP reviews suck! But why is not clear– @aclrollingreview.bsky.social provides *a lot* of guidance for how to review, but in that, first principles get lost. So @adamlopez.bsky.social and I have written down some of our thoughts on first principles.

MMLU-Redux just touched down at #NAACL2025! 🎉 Wish I could be there for our "Are We Done with MMLU?" poster today (9:00-10:30am in Hall 3, Poster Session 7), but visa drama said nope 😅 If anyone's swinging by, give our research some love! Hit me up if you check it out! 👋

Come say hi :)

Caught off-guard by the Llama 3.3 release? This is the loss of Llama-3.3-70B-Instruct (4bit quantized) on its own Twitter release thread. It really didn't like ' RL' (loss of 13.47) and wanted the text to instead go "... progress in online learning, which allows the model to adapt"

Do instruct models store factual associations differently than base models? 🤔 Doesn't look like it! When adapting ROME's causal tracing code to Llama 3.1 8B, the plots look very similar (base on top, instruct at the bottom). Note the larger sample size for instruct: If the "correct prediction" 1/3

How do LLMs learn to reason from data? Are they ~retrieving the answers from parametric knowledge🦜? In our new preprint, we look at the pretraining data and find evidence against this: Procedural knowledge in pretraining drives LLM reasoning ⚙️🔢 🧵⬇️

Hello to all #ICLR reviewers on #MLsky

1/2 The original 2022 ROME paper by Meng et al.: