👩‍💻 Running evals? Contribute your evaluation data to the community! Add your data to Every Eval Ever as part of the ACL Shared Task evalevalai.com/events/share... (Any eval data welcome - and we provide converters from lm-eval, Inspect, and HELM) Submission deadline: May 1, 2026

👶 BabyLM is back at EMNLP 2026! We are excited to announce that the 4th BabyLM Challenge & Workshop will once again bring together researchers interested in sample-efficient, developmentally plausible language modeling. @emnlpmeeting More in🧵

Agents should be general Why are we building code agents, CLI agents, browser agents separately? Why does adapting to a new benchmark take a month? Our collaboration brings diverse views, pros here cons in the paper & Your push back if I’m wrong Argument + paper link👇🧵 #MCP #ai #LLMs #agents 🤖📈🧠

Don't complain. Do it yourself. When the @eval-eval.bsky.social coalition started studying together what is broken in evaluation, I knew what we need to do. We need to digitize evals. How come every evaluation is reported differently? In a separate place? We built Every Eval Ever: 💪🤖📈🧠

LLM "robustness" is often treated like a mysterious, standalone capability. But what if it’s not? 🤔 Our new research shows robustness naturally appears when models truly understand a task - competence drives stability. More details in the thread 👇 🤖📈🧠 #AI arxiv.org/pdf/2602.03344

I just saw an LLM making a fluency mistake. How can that happen?! Something about long context?

"Kill your darlings, cut unless serving the story" One tip from the guide. If you needed a moment out of #ICML2026 writing and graphs, why not read some writing and figure making tips? docs.google.com/document/d/1...

"It is time to separate language from language models" The revelation keeps bugging me, so it also underlies the talk I just gave "multilingual?" Thought I'd briefly share the contents of the talk: 🤖📈🧠

From the conference that brought you the AC instead of rebuttal: Desk rejecting months after reviews were given. @iclr-conf.bsky.social - initiatives are good, but...

It'd be interesting to study how the substance of what functioning robots learn differs from LLMs. LLMs are based on language which in a sense is at least one level of abstraction up from experiential reality; language is humans' expression (a compression) of their collective experiences.

On the unexplained similarity across networks In behavior, order and weights, we keep seeing evidence that learning is more consistent than one might think. A walk through occurrences, my thoughts and the open question, why?! What's your hypothesis, missed papers and thoughts 🤖📈🧠 #AI

“Today, I have a vision, a vision of superintelligence from experience” Presented in his humble way, Rich Sutton shares his vision of what AI needs General, experiential, discovers its own abstractions and not bitter🤢 #NeurIPS2025 #NeurIPS 🤖📈🧠

LLMs do not learn from experience LLMs do not learn from explicit corrections LLMs do not learn from being told the answer LLMs do not learn from being shown how to solve it We study Machine Learning, these are opportunities! A gold mine of research.

"Hey dude, look. What is this button doing?" ⚡️BzZzZz⚡️ "Hey dude,..." Would you press the button again? Would an LLM? Evolving LLMs, diverse open LLMs, and their evaluation are on my mind. Before I share more, I encourage you to say hi here or in #NeurIPS 🤖📈🧠

Join us at NeurIPS 2025 for the MindGames Challenge Workshop! Explore theory of mind, game intelligence, and multi-agent LLMs in interactive game environments. 🗓 Sunday, December 7 ⏰ 8:00–10:45 AM 📍 San Diego Convention Center, Ballroom 6CF 🤖📈🧠

Quietly quietly Yuntian Deng released 2 million new chats of real humans with models (non-toxic), Kudo. There are now datasets of over 4.5m chats open for research and all in the same format (shareLM)! huggingface.co/datasets/sha... t\h @msheshera.bsky.social

001-110 101-? AAT-TTA TAT-? In context learning emerge outside language wonderful finding

The Golden pacifiers are ready See you soon in BabyLM (emnlp)

Introducing Global PIQA, a new multilingual benchmark for 100+ languages. This benchmark is the outcome of this year’s MRL shared task, in collaboration with 300+ researchers from 65 countries. This dataset evaluates physical commonsense reasoning in culturally relevant contexts.

🌍Introducing BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data! LLMs learn from vastly more data than humans ever experience. BabyLM challenges this paradigm by focusing on developmentally plausible data We extend this effort to 45 new languages!

With a fantastic team of international collaborators we have developed a pipeline for creating LM training data from resources that children are exposed to. We release this pipeline and welcome new contributions! Website: babylm.github.io/babybabellm/ Paper: arxiv.org/pdf/2510.10159

𝐃𝐨 𝐲𝐨𝐮 𝐫𝐞𝐚𝐥𝐥𝐲 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐬𝐞𝐞 𝐰𝐡𝐚𝐭 𝐦𝐮𝐥𝐭𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐞𝐟𝐟𝐨𝐫𝐭 𝐥𝐨𝐨𝐤𝐬 𝐥𝐢𝐤𝐞? 🇨🇳🇮🇩🇸🇪 Here’s the proof! 𝐁𝐚𝐛𝐲𝐁𝐚𝐛𝐞𝐥𝐋𝐌 is the first Multilingual Benchmark of Developmentally Plausible Training Data available for 45 languages to the NLP community 🎉 arxiv.org/abs/2510.10159

✨ The schedule for our INTERPLAY workshop at COLM is live! ✨ 🗓️ October 10th, Room 518C 🔹 Invited talks from @sarah-nlp.bsky.social John Hewitt @amuuueller.bsky.social @kmahowald.bsky.social 🔹 Paper presentations and posters 🔹 Closing roundtable discussion. Join us in Montréal! @colmweb.org

LLM, VLMs, ... can compress data 3x over JPEG\PNG etc. 6x Zlib, gzip etc. How? We all know they provide a probability over data, which is all classical compression needs (arithmetic coding, see below) Understanding is compressing, but this time not by the weights themselves 🤖📈🧠 #AI #compress #data

LLMs are trained to mimic a “true” distribution—their reducing cross-entropy then confirms they get closer to this target while training. Do similar models approach this target distribution in similar ways, though? 🤔 Not really! Our new paper studies this, finding 4-convergence phases in training 🧵

Employing mechanistic interpretability to study how models learn, not just where they end up 2 papers find: There are phase transitions where features emerge and stay throughout learning 🤖📈🧠 alphaxiv.org/pdf/2509.17196 @amuuueller.bsky.social @abosselut.bsky.social alphaxiv.org/abs/2509.05291

Helpfulness is what we are after, and we test it by asking humans for preferences, or reward models. and they fail😆 They show that humans are bad at predicting what is helpful, so are reward models (all close to chance). Reward models don't even predict what helps LLMs RL🤔 🤖📈🧠 #AI #LLM

Good luck with the @iclr_conf writing Know anyone who needs tips? Want a graph checklist? Know any good tips you wanna add? The writing guide: docs.google.com/document/d/1...

The most expensive part of training is the data, not the compute Nikhil Kandpal & Colin Raffel calculate a really low bar for how much it would cost to produce LLM training data with 3.8$\h Well, several scales more than the compute. Luckily (?), companies don't pay for the data 🤖📈🧠

Anyone willing to make an emergency review for babyLM?

A dataset of ancient Chinese writings to study with LLMs, including 170K sentences for pretraining With 10K words, mapping to modern word (when applicable) There are so many fascinating questions out there www.arxiv.org/abs/2508.15791

Why is it hard to predict downstream scores from pretraining? For many reasons such as domain, mismatch between current abilities and what post training unfolds, "emergence" etc. A big factor is that next token prediction != choice comparison != accuracy www.alphaxiv.org/abs/2406.04391

The right path is unclear; maybe it doesn't even exist. Still, LLMs appear to consistently follow the values of secular\rational people who strive for self-expression (sounds like me😅) To show it they collect and release 200K human-model chats+feedback, 5 languages and 21 LLMs 🤖📈🧠

Pressing matters in evaluation from GEMs talk Remember, exciting questions drive science, exciting answers follow. Setting the right goal may make all their sota chasing worthwhile. Make an insightful dataset, lead by evaluation 🤖📈🧠

The talk for our work on Retrospective Learning from Interactions, which will be in ACL (once I figure out how to squeeze it shorter) Gist: autonomous post-training from conversational signals for LLM bootstrapping ... look ma, no annotations! no hand-holding! 🙌📈🚀 www.youtube.com/watch?v=qW8S...

ICML proposes we use sophisticated jailbreaks in our papers? Ones that trick the reviewers but do not raise our scores? Proposals?

Can LLMs learn social skills by playing games? A blogpost on human-model interaction, games, training and testing LLMs research.ibm.com/blog/LLM-soc... 🤖📈🧠

We don't understand loss spikes, that's clear. We've learned recently that data deterministically makes spikes regardless of optimizer. What did we see when we stopped pretraining, and then continue? A huge spike and never a recovery? Why? Apparently the momentum matters, a lot. 🤖📈🧠

Eu calls for evaluator experts to counsel about how to evaluate models (can't belong to a training company, smart) digital digital-strategy.ec.europa.eu/en/news/comm... #safety #eticalAI #AI 🤖📈🧠 Tag people who should know or should pass the knowledge

The ML\LLM AI world is overcrowded✅ and it's hard to find majestic new projects❌ Here is (updating) list of thoughts, premature ideas, and research directions that I cannot pursue alone, but I believe can make radical changes Please fight me, discuss, and ask Disrrruption, aye?🏴‍☠️ 🤖📈🧠

Surprisingly good tokenization workshop, resurfaced thoughts:🧠📈🤖 Why isn't tokenization learned? Can we do an evolutionary algorithm, or train a tokenization scheme on the pretrained or meta learn on something fast-pretrain (loss at the beginning of training?) Let's discuss👇

Do models really understand humans (ToM)? Can your model do it? Beat them at the neurips Competition. 🤖📈🧠

How do you test a new model? What's super challenging for models? The @kaggle.com team are starting a new way to crowd evaluate models, create your own benchmarks.🚀 (i'll tell you mine🧵 if you tell me yours) 🤖📈🧠 kaggle-task-generator-564531943159.us-west1.run.app

Models self-teach slightly more complex skills than the teacher knows And they can do so again and again until it is not just slightly. My questions:🧵

Teaching ChatGPT to tell everyone how handsome I am

Imagine a hacker causing ChatGPT to genuinely believe the stock market is crashing. No breach, no code, all an attacker needs is a malicious prompt and clicking 👍to change the model's weights. Our research paper demonstrates a vulnerability in LLMs RLHF mechanism: arxiv.org/abs/2507.02850 📈🤖🧠

The Swedish National Archives shared 1.2m images of handwritten documents from 1600-1900+ In addition, they share a model for writing to text Such marvelous data opportunities, kudos 📈🤖🧠 medium.com/@linnea.karl... Example: sok.riksarkivet.se/om-soktjanst... huggingface.co/datasets/Rik...

For the past ~2 months we have been working on training reasoning models on TextArena.ai games. Soon, a paper introducing what we think is a promising new paradigm (+ scaling and analysis). However, to get more feedback on the structure and implementation, we want to open-source the code now.

"Over four months, LLM users consistently underperformed at neural, linguistic, and behavioral levels. These results raise concerns about the long-term educational implications of LLM reliance and underscore the need for deeper inquiry into AI's role in learning." arxiv.org/abs/2506.08872

🚀 Technical practitioners & grads — join to build an LLM evaluation hub! Infra Goals: 🔧 Share evaluation outputs & params 📊 Query results across experiments Perfect for 🧰 hands-on folks ready to build tools the whole community can use Join the EvalEval Coalition here 👇 forms.gle/6fEmrqJkxidy...