For your Embodied AI task you want a recurrent model with constant complexity per step, but you don't want to lose the power of transformers (which store the full obs history and attend to it)? Do not despair, we have your back. We distill transformers into recurrent transformers 1/8

New review with Cheng Xue at U Chicago @cxue.bsky.social in Trends [email protected]! We discuss the neural geometry of task-dependent computation: disentangled encoding, RNN modeling, switch cost, etc. www.cell.com/trends/neuro...

A long read about the state of AI and mathematics. davidbessis.substack.com/p/the-fall-o...

🧵 New preprint led by @bingbrunton.bsky.social, @elliottabe.bsky.social, @lawrencehu.bsky.social We gave a worm brain control of a fly body and it walked What did we learn? Nothing, other than deep reinforcement learning is effective We call it the digital sphinx www.biorxiv.org/content/10.6...

www.percepta.ai/blog/can-llm... As a research lark at Percepta, Christos embedded a computer into an LLM, showed that it could solve the hardest Sudokus, and then as a side bonus built an exponentially faster attention

Bullshit Bench V2 new: 100 questions across several domains - Anthropic & Qwen still on top - Reasoning seems to hurt - New models are *not* better than old (except Claude) - Seems to be independent of domain github.com/petergpt/bul...

Sakana has developed a way to, if I understand correctly, instantly generate LORAs on demand from long texts or documents arxiv.org/abs/2506.06105 arxiv.org/abs/2602.15902

Trump has been in office for one year. We at @nature.com did a deep dive looking at the administration's disruption of science in numbers. Take a look—the numbers are staggering. By me, @dangaristo.bsky.social, Jeff Tollefson, @kimay.bsky.social, & help from @noamross.net @scott-delaney.bsky.social

This is the most astonishing graph of what the Trump regime has done to US science. They have destroyed the federal science workforce across the board. The negative impacts on Americans will be felt for generations, and the US might never be the same again. www.nature.com/immersive/d4...

One of my favorite findings: Positional embeddings are just training wheels. They help convergence but hurt long-context generalization. We found that if you simply delete them after pretraining and recalibrate for <1% of the original budget, you unlock massive context windows. Smarter, not harder.

Oh wow, deepseek is starting to make serious progress on LLMs that offload memory to external storage: github.com/deepseek-ai/...

Excited to see our paper with @mwcole.bsky.social finally out in peer-reviewed form @natcomms.nature.com! We examine how the human brain learns new tasks and optimizes representations over practice…1/n

Did you know that AI can figure out its own way to learn, and that its way is better than one designed by humans? Read more in a @nature.com N&V (and the original paper is in the comment) 🧪 www.nature.com/articles/d41...

Our work with @pawa-pawa.bsky.social is out in Nature Machine Intelligence! The choice of activation function affects the representations, dynamics, and circuit solutions that emerge in RNNs trained on cognitive tasks. Activation matters! www.nature.com/articles/s42...

(repost welcome) The Generative Model Alignment team at IBM Research is looking for next summer interns! Two candidates for two topics 🍰Reinforcement Learning environments for LLMs 🐎Speculative and non-auto regressive generation for LLMs interested/curious? DM or email [email protected]

Michael X Cohen on why he left academia/neuroscience. mikexcohen.substack.com/p/why-i-left...

Nature research paper: Arousal as a universal embedding for spatiotemporal brain dynamics go.nature.com/4nMUgYz

Lab’s latest is out in Imaging Neuroscience, led by Kirsten Peterson: “Regularized partial correlation provides reliable functional connectivity estimates while correcting for widespread confounding”, where we demonstrate a major improvement to standard fMRI functional connectivity (correlation) 1/n

What complexity of algorithms can AI compute? In a new paper with colleagues at IBM Research, we explore how circuit complexity theory can help quantify the degree of algorithmic generalization in AI systems. www.nature.com/articles/s42... @natmachintell.nature.com #ML #AI #MLSky 1/n

Mental health research is at a turning point—breakthroughs can transform lives, but only with bold action, investment, and open collaboration. The time for action is now. Read our full statement here: childmind.org/blog/can-sci...

Out today in Nature Machine Intelligence! From childhood on, people can create novel, playful, and creative goals. Models have yet to capture this ability. We propose a new way to represent goals and report a model that can generate human-like goals in a playful setting... 1/N

New preprint! Ziyan and I explore how task order impacts continual learning in neural networks and how to optimize it. Our analysis highlights two key principles for better task sequencing. Check it out: arxiv.org/pdf/2502.03350

The entire website for the NIH Office of Research on Women's Health (ORWH) is very nearly stripped bare. This is so, so devastating. orwh.od.nih.gov/research/fun...

New paper out! 🚨 📰 With @batuhanerkat.bsky.social, John McClure, @hussainyk1.bsky.social, @polacklab.bsky.social we reveal how discretized representations in V1 predict suboptimal orientation discrimination. 🧪🧠🐭 This work reconciles neuro and psychometric curves www.nature.com/articles/s41...

New paper in @brain1878.bsky.social: Healthy people under S-ketamine, an NMDAR antagonist, and people living with schizophrenia, a disorder associated with NMDAR hypofunction, spend more time in an external mode of perception - where noisy sensory signals override knowledge about the world.

The origin of color categories | PNAS www.pnas.org/doi/10.1073/...

Check our latest in which we leverage shape metrics to compare neural geometry across regions, sessions or subjects and how their differences predict behavior. w/ Nejatbakhsh, Duong, @sarah-harvey.bsky.social, Brincat, @siegellab.bsky.social, @earlkmiller.bsky.social & @itsneuronal.bsky.social

Paper shows very small LLMs can match or beat larger ones through 'deep thinking' - evaluating different solution paths - and other tricks. Their 7B model beats o1-preview on complex math by exploring 64 different solutions & picking the best one. Test-time compute paradigm seems really fruitful.

New results for a new year! “Linking neural population formatting to function” describes our modern take on an old question: how can we understand the contribution of a brain area to behavior? www.biorxiv.org/content/10.1... 🧠👩🏻‍🔬🧪🧵 #neuroskyence 1/

And relatedly, Felix wrote a good piece on the stress and anxiety currently affecting many people who work in AI due to the current climate in the industry: docs.google.com/document/d/1... If only more folks in AI were gentle and introspective like this...

What was the most important machine learning paper in 2024? My Famous Deep Learning Papers list (that I use in teaching) does not include any new ideas from the last year. papers.baulab.info Which single new paper would you add?

Some of my thoughts on OpenAI's o3 and the ARC-AGI benchmark aiguide.substack.com/p/did-openai...

📌 Poster Session: ⏰ When: TODAY, Thu, Dec 12, 4:30 p.m. – 7:30 p.m. PST 📍 Where: East Exhibit Hall A-C, #3705 📄 What: Geometry of Naturalistic Object Representations in Recurrent Neural Network Models of Working Memory Hope to see you there! @bashivan.bsky.social @takuito.bsky.social

🚨We're very excited to share our latest study, by Pablo Diego and team: "A polar coordinate system represents syntax in large language models", 📄: Paper arxiv.org/abs/2412.05571 🪧: Poster tomorrow: neurips.cc/virtual/2024... 🧵: Thread 👇

Just published🔈 "Structurally informed models of directed brain connectivity" Read: rdcu.be/d3dC4 We review how structural connectivity constrains directed connectivity models 🧠 Lead by @matthewdgreaves.bsky.social w/ @novelli-leo.bsky.social, @sinamansourl.bsky.social and Andrew Zalesky

hey -- i'm hiring a postdoc! the ad will be up shortly, but looking for someone with network neuroscience experience (very broadly). the position isn't tied to any specific project/grant, so lots of flexibility in terms of what you'd actually *do*. hmu if you might be interested/want to learn more!

“Cognitive flexibility as the shifting of brain network flows by flexible neural representations”, a solo paper by yours truly, making the case that brain activity flow shifts are essential to mental flexibility (and quite interesting too!) Open access: www.sciencedirect.com/science/arti...

Our new NeurIPS paper on naturalistic representations in dynamic WM models, led by @xiaoxuanlei.bsky.social and @bashivan.bsky.social Thread by Xiaoxuan 👇

JAMA Psychiatry A Dynamical Systems View of Psychiatric Disorders—Theory A Review jamanetwork.com/journals/jam...

Lab’s latest at PLOS Comp Biol, led by Carrisa Cocuzza: “Distributed network flows generate localized category selectivity in human visual cortex”. This one changed how I think the brain works! Even "localized" functions are likely generated by distributed processes doi.org/10.1371/jour...

Practice Reshapes the Geometry and Dynamics of Task-tailored Representations https://www.biorxiv.org/content/10.1101/2024.09.12.612718v1

This new article perfectly concludes my time & lessons in Jim’s lab.@JamesJDiCarlo and I propose +review SMART models of object recognition ✅ Sensory computable ✅ Mechanistic ✅ Anatomically Referenced ✅ Testable Coming in Annual Reviews 2024 Preprint: bit.ly/3tk7u8D

www.nature.com/articles/s41... Lake & Baroni "Fodor and Pylyshyn famously argued that artificial neural networks ... are ... not viable models of the mind.... Here we [show] that neural networks can achieve human-like systematicity when optimized for their compositional skills."

From lazy to rich to exclusive task representations in neural networks and neural codes doi.org/10.1016/j.co... #neuroscience

NeuroImage Spatially heterogeneous structure-function coupling in haemodynamic and electromagnetic brain networks www.sciencedirect.com/science/arti...

How does the brain represent multiple different things at once in a single population of neurons? @justfineneuro.bsky.social, @benhayden.bsky.social, B Ebitz, M Yoo, and I show that it uses semi-orthogonal subspaces for each item. Preprint here: arxiv.org/abs/2309.07766 Clouds below! (1/n)

Semi-orthogonal subspaces for value mediate a tradeoff between binding and generalization arxiv.org/abs/2309.07766