If you're in Berkeley or like a nice streamed talk, I'm about to give a talk at the Simons Institute today: "You Know It Or You Don’t: Compositionality and Phase Transitions in LMs". Tune in at 4PM pacific!

What to know about DeepSeek youtu.be/0eMzc-WnBfQ?... In which we attempt to figure out MoE, o1, scaling, tech reporting, modern semiconductors, microeconomics, and international geopolitics.

For reasons, I find myself thinking a lot about the history of US/USSR Cold War science, particularly in applied math. Does anyone have a recommendation for a good book on this topic?

10 short videos about LLM infrastructure to help you appreciate Pages 12-18 of the DeepSeek-v3 paper (arxiv.org/abs/2412.19437) www.youtube.com/watch?v=76gu...

I tried reading this book, and I was just shocked at how little insight it had, and it's sheer inability to focus. The fact that it is being recommended to policy makers... www.gatesnotes.com/The-Coming-W...

I'm going to do a live coding stream for the next couple of hours. We'll start by running through some WebGPU tutorials. Can also talk about some AI stuff. www.youtube.com/watch?v=sqKq...

We outperform Llama 70B with Llama 3B on hard math by scaling test-time compute 🔥 How? By combining step-wise reward models with tree search algorithms :) We're open sourcing the full recipe and sharing a detailed blog post 👇

As coding LLMs get faster at inference, iterating verification-in-the-loop tests becomes the bottleneck for coding agents. Probably need quite different programming systems for these settings, or even things like "batchable" runtimes, whatever that means.

We organised a lively poster fest with many students rehearsing for the upcoming @neuripsconf.bsky.social next week and others discussing their cool works! Thanks to #GAIL, the #Generative #AI lab in #Edinburgh for sponsoring the event!

I wanted to make my first post about a project close to my heart. Linear algebra is an underappreciated foundation for machine learning. Our new framework CoLA (Compositional Linear Algebra) exploits algebraic structure arising from modelling assumptions for significant computational savings! 1/4

Proud of my amazing colleague @sashamtl.bsky.social for her much deserved recognition on advancing the science of AI energy use. BBC100: www.bbc.co.uk/news/resourc... Fast Company: www.fastcompany.com/91233692/why... Sasha has been working tirelessly moving things fwd--endurance & brilliance in one.

NEW: we have an exciting opportunity for a tenure-track professor at the #KempnerInstitute and the John A. Paulson School of Engineering and Applied Sciences (SEAS). Read the full description & apply today: academicpositions.harvard.edu/postings/14362 #ML #AI

We're hiring another predoctoral researcher for my team at Ai2/OLMo next year. The goal of this position is to mentor and grow future academic stars of NLP/AI over 1-2 years before grad school. This ends up being people done with BS or MS who want to continue to a PhD soon. https://buff.ly/49nuggo

Is there a community that writes RL-first programming languages? Something like (Num)Pyro that takes seriously the idea of separating the policy specification from the learning process.

ICYMI, @srushnlp.bsky.social recently gave a nice talk speculating about the methods/data used to train OpenAI's o1 model. The key idea seems to be scaling up chain-of-thought (CoT) generation using auxiliary verifier models that can give feedback on the correctness of the generation. 1/N

My deep learning course at the University of Geneva is available on-line. 1000+ slides, ~20h of screen-casts. Full of examples in PyTorch. fleuret.org/dlc/ And my "Little Book of Deep Learning" is available as a phone-formatted pdf (nearing 700k downloads!) fleuret.org/lbdl/

A starter pack for research on Formal Languages and Neural Networks! go.bsky.app/eKG7Ua

Hello Bluesky! 🔵 We start our account by having our third guess for Ask Me Anything session #3DV2025AMA! Noah Snavely @snavely.bsky.social from Cornell & Google DeepMind! 🌟 🕒 You have now 24 HOURS to ask him anything — drop your questions in the comments below! Keep it engaging but respectful!

Rare personal tweet: Subletting our furnished apartment in Brooklyn for the spring at a significant discount. It's quite nice and in a fun location. under price. Email me know if you are interested, I will send pictures.

Several incredible NeurIPS tutorials this year. Worth navigating through the Swifties.

Another nano gem from my amazing student Piotr Nawrot! A repo & notebook on sparse attention for efficient LLM inference: github.com/PiotrNawrot/... This will also feature in my #NeurIPS 2024 tutorial "Dynamic Sparsity in ML" with André Martins: dynamic-sparsity.github.io Stay tuned!

I've spent the last two years scouring all available resources on RLHF specifically and post training broadly. Today, with the help of a totally cracked team, we bring you the fruits of that labor — Tülu 3, an entirely open frontier model post training recipe. We beat Llama 3.1 Instruct. Thread.

Discrete diffusion has become a very hot topic again this year. Dozens of interesting ICLR submissions and some exciting attempts at scaling. Here's a bibliography on the topic from the Kuleshov group (my open office neighbors). github.com/kuleshov-gro...

A disproportionate number of sucessful CS academics have some intense cardio hobby. Took me some years to understand.

I did a starter pack of people in New York (City) working on ML/AI. Please distribute and feel free to self nominate! go.bsky.app/BoEtagz

hard for me to get engagement on this platform but it’s ok I’m just waiting for all the AI researchers to show up so I can go mega viral saying shit like “who called it Xavier Initialization and not Weighting For Glorot”

I keep reading interviews about how impressively smart and accomplished the AlphaProof team is. But I would be way more impressed if they were just like "we're super bad at math, who cares".

Welcome!

Guest Video: Should I do a Postdoc? - Niloofar Mireshghallah It's the time of the year when a lot of CS PhDs are thinking about next steps in the process. Niloofar gives a thoughtful, personal view on postdoctoral life. www.youtube.com/watch?v=O7ps...

Talk: Speculations on Test-Time Scaling A tutorial on the technical aspects behind OpenAI's o1 and open research questions in this space. youtu.be/6PEJ96k1kiw Slides+bibliography: github.com/srush/awesom...

I'll be at EMNLP this week. I would love to chat. I'm interested in learning for reasoning, alternative architectures, embedding models, text diffusion, and where NLP is going as a field. calendly.com/srush-resear...

"The Elements of Differentiable Programming" link: arxiv.org/abs/2403.14606 Basically: "autodiff - it's everywhere! what is it, and how do you use it?" seems like a good resource for anyone interested in data science, machine learning, "ai," neural nets, etc #blueskai #stats #mlsky

Heard we are moving NLP twitter over here now. Seems like a good time, I'll try to cross post.

Talk: Inverting Language Models youtube.com/watch?v=lguT... Techniques for extracting text from vector databases and prompts from LLM APIs.

Introducing COLM (colmweb.org) the Conference on Language Modeling. A new research venue dedicated to the theory, practice, and applications of language models. Submissions: March 15 (it's pronounced "collum" )

Autodiff Puzzle (v0.2, github.com/srush/Autodi...) - How come my Calculus teacher never taught me the derivative of sort()? 20 one-line puzzles for differentiating all the things.