Why does AI sometimes fail to generalize, and what might help? In a new paper (arxiv.org/abs/2509.16189), we highlight the latent learning gap — which unifies findings from language modeling to agent navigation — and suggest that episodic memory complements parametric learning to bridge it. Thread:

How can an imitative model like an LLM outperform the experts it is trained on? Our new COLM paper outlines three types of transcendence and shows that each one relies on a different aspect of data diversity. arxiv.org/abs/2508.17669

In neuroscience, we often try to understand systems by analyzing their representations — using tools like regression or RSA. But are these analyses biased towards discovering a subset of what a system represents? If you're interested in this question, check out our new commentary! Thread:

Get ready to enter the simulation... Genie 3 is a new frontier for world models: its environments remain largely consistent for several minutes, with visual memory extending as far back as 1min. These limitations will only decrease with time. Welcome to the future.🙌 deepmind.google/discover/blo...

Looking forward to attending CogSci this week! I'll be giving a talk (see below) at the Reasoning Across Minds and Machines workshop on Wednesday at 10:25 AM, and will be around most of the week — feel free to reach out if you'd like to meet up!

This summer my lab's journal club somewhat unintentionally ended up reading papers on a theme of "more naturalistic computational neuroscience". I figured I'd share the list of papers here 🧵:

Does vision training change how language is represented and used in meaningful ways?🤔The answer is a nuanced yes! Comparing VLM-LM minimal pairs, we find that while the taxonomic organization of the lexicon is similar, VLMs are better at _deploying_ this knowledge. [1/9]

Quick thread on the recent IMO results and the relationship between symbol manipulation, reasoning, and intelligence in machines and humans:

Excited to share a new project spanning cognitive science and AI where we develop a novel deep reinforcement learning model---Multitask Preplay---that explains how people generalize to new tasks that were previously accessible but unpursued.

Exciting new preprint from the lab: “Adopting a human developmental visual diet yields robust, shape-based AI vision”. A most wonderful case where brain inspiration massively improved AI solutions. Work with @zejinlu.bsky.social @sushrutthorat.bsky.social and Radek Cichy arxiv.org/abs/2507.03168

Thrilled to see our TinyRNN paper in @nature! We show how tiny RNNs predict choices of individual subjects accurately while staying fully interpretable. This approach can transform how we model cognitive processes in both healthy and disordered decisions. doi.org/10.1038/s415...

Experimentology is out today!!! A group of us wrote a free online textbook for experimental methods, available at experimentology.io - the idea was to integrate open science into all aspects of the experimental workflow from planning to design, analysis, and writing.

Really nice analysis!

Humans and animals can rapidly learn in new environments. What computations support this? We study the mechanisms of in-context reinforcement learning in transformers, and propose how episodic memory can support rapid learning. Work w/ @kanakarajanphd.bsky.social : arxiv.org/abs/2506.19686

I'm excited about this TICS Opinion with @yngwienielsen.bsky.social, challenging the view that structural priming—the tendency to reuse a recent syntactic structure—provides evidence for the psychological reality of grammar-based constituent structure. authors.elsevier.com/a/1lIFK4sIRv... 🧵1/4

What makes humans similar or different to AI? In a paper out in @natmachintell.nature.com led by @florianmahner.bsky.social & @lukasmut.bsky.social, w/ Umut Güclü, we took a deep look at the factors underlying their representational alignment, with surprising results. www.nature.com/articles/s42...

LLMs excel at finding surprising “needles” in very long documents, but can they detect when information is conspicuously missing? 🫥AbsenceBench🫥 shows that even SoTA LLMs struggle on this task, suggesting that LLMs have trouble perceiving “negative spaces”. Paper: arxiv.org/abs/2506.11440 🧵[1/n]

New and improved version of our recent preprint on how to build generalizable models & theories in cognitive science by (in part) drawing on lessons and tools from AI!

ACL paper alert! What structure is lost when using linearizing interp methods like Shapley? We show the nonlinear interactions between features reflect structures described by the sciences of syntax, semantics, and phonology.

A string may get 17 times less probability if tokenised as two symbols (e.g., ⟨he, llo⟩) than as one (e.g., ⟨hello⟩)—by an LM trained from scratch in each situation! Our new ACL paper proposes an observational method to estimate this causal effect! Longer thread soon!

Transformer-based neural networks achieve impressive performance on coding, math & reasoning tasks that require keeping track of variables and their values. But how can they do that without explicit memory? 📄 Our new ICML paper investigates this in a synthetic setting! 🎥 youtu.be/Ux8iNcXNEhw 🧵 1/13

🎥 The recording of the first ELLISxUniReps Speaker Series session with Andrew Lampinen @lampinen.bsky.social and Jack Lindsey is now available here: www.youtube.com/watch?v=eXE0... Next appointment: 20th June 2025 – 16:00 CEST on Zoom 🔵 Keynote: Mariya Toneva (MPI) 🔴 Flash: Lenka Tětková (DTU)

🚨 New pre-print! (Well, new & much improved version in any case.) 🚨 If you're interested in LLM post-training techniques and in how to make LLMs better "language users", read this thread, introducing the "LM Playpen".

📣 Starting in less than one hour! Join at: ethz.zoom.us/j/66426188160 at 4:30 pm CEST

New preprint! 🚨→ Determinants of Visual Ambiguity Resolution. A new work with @ortiztudela.bsky.social @jvoeller.bsky.social @martinhebart.bsky.social and @gonzalezgarcia.bsky.social We created ~2k images and collected ~100k responses to study visual ambiguity. www.biorxiv.org/content/10.1...

What are the organizing dimensions of language processing? We show that voxel responses during comprehension are organized along 2 main axes: processing difficulty & meaning abstractness—revealing an interpretable, topographic representational basis for language processing shared across individuals

New preprint alert! We often prompt ICL tasks using either demonstrations or instructions. How much does the form of the prompt matter to the task representation formed by a language model? Stick around to find out 1/N

Excited to share a new preprint w/ @michael-lepori.bsky.social & Michael Franke! A dominant approach in AI/cogsci uses *outputs* from AI models (eg logprobs) to predict human behavior. But how does model *processing* (across layers in a forward pass) relate to human real-time processing? 👇 (1/12)

🎉 The first @ellis.eu × Unireps Speaker Series (Reading Group associated with Ellis PhD & Postdoc Program) session is happening soon! 🔜 🗓️ When: 29th May 2025 – 16:30 CET 📍 Where: ethz.zoom.us/j/66426188160 🎙️ Who: Andrew Lampinen @lampinen.bsky.social (Keynote) & Jack Lindsey (Flash Talk)

It was an honor to speak at the @kempnerinstitute.bsky.social on how some of our recent works can be interpreted as an application of rational analysis to language model behaviors — seeing both their successes and failures through the lens of adaptation to the properties of their training data.

How do language models generalize from information they learn in-context vs. via finetuning? In arxiv.org/abs/2505.00661 we show that in-context learning can generalize more flexibly, illustrating key differences in the inductive biases of these modes of learning — and ways to improve finetuning. 1/

Had fun talking at the Spurious Correlations & Shortcut Learning at ICLR! One example I brought up, which I think provides an uncommon perspective: a case where spurious shortcuts can improve generalization... even to out-of-distribution sets where the spurious feature doesn't generalize! Thread:

Some important points here — the way I'd put it is that the gap between even 1) AI being better than many experts at certain well-specified math/AI research tasks and 2) progress in the field accelerating by orders of magnitude is still huge.

💡 New ICLR paper! 💡 "On Linear Representations and Pretraining Data Frequency in Language Models": We provide an explanation for when & why linear representations form in large (or small) language models. Led by @jackmerullo.bsky.social, w/ @nlpnoah.bsky.social & @sarah-nlp.bsky.social

Excited to head to ICLR this week! Feel free to reach out if you'll be there and would like to chat. I'll be giving a talk at the workshop on spurious correlations and shortcut learning, and we'll have some posters scattered across various other workshops.

Applications are open for the 2025 Flatiron Institute Junior Theoretical Neuroscience Workshop! A two-day workshop 7/10-7/11 in NYC for PhD students and postdocs. All travel paid. Apply by April 14th.🧠🗽🧑‍🔬http://jtnworkshop2025.flatironinstitute.org/ @flatironinstitute.org @simonsfoundation.org

This past Friday I successfully defended my PhD 🎉 🎊👨🏻‍🎓 What a journey it was! 4.5 years of many ups and many downs. Can’t believe it’s over. I am still processing… Special thanks to my wonderful committee KR Müller, @martinhebart.bsky.social, @ericschulz.bsky.social, and @scychan.bsky.social!

Some fascinating details of the dynamics and mechanisms of how models acquire factual knowledge!

This exactly. Reading the lit as a skill is very much about learning to ignore junk. It’s extremely bad that this is true but it is.

🚀 "Multilingual" LLMs are really just clusters of monolingual ones! They might know a Brazilian 🇧🇷 brewery—but only in Portuguese With ECLEKTIC, you can now test this. The challenge? Making them truly multilingual. alphaxiv.org/pdf/2502.21228 📈🤖 🧵⬇️ #LLM #ai #genAI #nlp

I think there's an implication of this that will matter beyond language differences, for any attempt to use models in "cultural simulation" — which is that you can never *assume* linguistic behaviors are mediated by some deeper representation that translates across genres, registers, etc.

My latest paper is just out on arXiv: : "Palatable Conceptions of Disembodied Being: Terra Incognita in the Space of Possible Minds". 🧵1/n

my paper with max, @maxkw.bsky.social, tuomas, and @fierycushman.bsky.social out in cognition at long last www.sciencedirect.com/science/arti... We explain why humans and successful AI planners both fail on a certain kind of problem that we might describe as requiring insight or creativity

New preprint “Monkey See, Model Knew: LLMs accurately predict visual responses in humans AND NHPs” Led by Colin Conwell with @emaliemcmahon.bsky.social Akshay Jagadeesh, Kasper Vinken @amrahs-inolas.bsky.social @jacob-prince.bsky.social George Alvarez @taliakonkle.bsky.social & Marge Livingstone 1/n

AI models are fascinating, impressive, and sometimes problematic. But what can they tell us about the human mind? In a new review paper, @noahdgoodman.bsky.social and I discuss how modern AI can be used for cognitive modeling: osf.io/preprints/ps...

1/13 New Paper!! We try to understand why some LMs self-improve their reasoning while others hit a wall. The key? Cognitive behaviors! Read our paper on how the right cognitive behaviors can make all the difference in a model's ability to improve with RL! 🧵

New preprint! In arxiv.org/abs/2502.20349 “Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior” we synthesize AI & cognitive science works to a perspective on seeking generalizable understanding of cognition. Thread:

Ever looked at LLM skill emergence and thought 70B parameters was a magic number? Our new paper shows sudden breakthroughs are samples from bimodal performance distributions across seeds. Observed accuracy jumps abruptly while the underlying accuracy DISTRIBUTION changes slowly!

What should count as a good model of intelligence? AI is advancing rapidly, but how do we know if it captures intelligence in a scientifically meaningful way? We propose the *NeuroAI Turing Test*—a benchmark that evaluates models based on both behavior and internal representations.