The Golden pacifiers are ready See you soon in BabyLM (emnlp)

Introducing Global PIQA, a new multilingual benchmark for 100+ languages. This benchmark is the outcome of this year’s MRL shared task, in collaboration with 300+ researchers from 65 countries. This dataset evaluates physical commonsense reasoning in culturally relevant contexts.

🌍Introducing BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data! LLMs learn from vastly more data than humans ever experience. BabyLM challenges this paradigm by focusing on developmentally plausible data We extend this effort to 45 new languages!

With a fantastic team of international collaborators we have developed a pipeline for creating LM training data from resources that children are exposed to. We release this pipeline and welcome new contributions! Website: babylm.github.io/babybabellm/ Paper: arxiv.org/pdf/2510.10159

𝐃𝐨 𝐲𝐨𝐮 𝐫𝐞𝐚𝐥𝐥𝐲 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐬𝐞𝐞 𝐰𝐡𝐚𝐭 𝐦𝐮𝐥𝐭𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐞𝐟𝐟𝐨𝐫𝐭 𝐥𝐨𝐨𝐤𝐬 𝐥𝐢𝐤𝐞? 🇨🇳🇮🇩🇸🇪 Here’s the proof! 𝐁𝐚𝐛𝐲𝐁𝐚𝐛𝐞𝐥𝐋𝐌 is the first Multilingual Benchmark of Developmentally Plausible Training Data available for 45 languages to the NLP community 🎉 arxiv.org/abs/2510.10159

✨ The schedule for our INTERPLAY workshop at COLM is live! ✨ 🗓️ October 10th, Room 518C 🔹 Invited talks from @sarah-nlp.bsky.social John Hewitt @amuuueller.bsky.social @kmahowald.bsky.social 🔹 Paper presentations and posters 🔹 Closing roundtable discussion. Join us in Montréal! @colmweb.org

LLM, VLMs, ... can compress data 3x over JPEG\PNG etc. 6x Zlib, gzip etc. How? We all know they provide a probability over data, which is all classical compression needs (arithmetic coding, see below) Understanding is compressing, but this time not by the weights themselves 🤖📈🧠 #AI #compress #data

LLMs are trained to mimic a “true” distribution—their reducing cross-entropy then confirms they get closer to this target while training. Do similar models approach this target distribution in similar ways, though? 🤔 Not really! Our new paper studies this, finding 4-convergence phases in training 🧵

Employing mechanistic interpretability to study how models learn, not just where they end up 2 papers find: There are phase transitions where features emerge and stay throughout learning 🤖📈🧠 alphaxiv.org/pdf/2509.17196 @amuuueller.bsky.social @abosselut.bsky.social alphaxiv.org/abs/2509.05291

Helpfulness is what we are after, and we test it by asking humans for preferences, or reward models. and they fail😆 They show that humans are bad at predicting what is helpful, so are reward models (all close to chance). Reward models don't even predict what helps LLMs RL🤔 🤖📈🧠 #AI #LLM

Good luck with the @iclr_conf writing Know anyone who needs tips? Want a graph checklist? Know any good tips you wanna add? The writing guide: docs.google.com/document/d/1...

The most expensive part of training is the data, not the compute Nikhil Kandpal & Colin Raffel calculate a really low bar for how much it would cost to produce LLM training data with 3.8$\h Well, several scales more than the compute. Luckily (?), companies don't pay for the data 🤖📈🧠

Anyone willing to make an emergency review for babyLM?

A dataset of ancient Chinese writings to study with LLMs, including 170K sentences for pretraining With 10K words, mapping to modern word (when applicable) There are so many fascinating questions out there www.arxiv.org/abs/2508.15791

Why is it hard to predict downstream scores from pretraining? For many reasons such as domain, mismatch between current abilities and what post training unfolds, "emergence" etc. A big factor is that next token prediction != choice comparison != accuracy www.alphaxiv.org/abs/2406.04391

The right path is unclear; maybe it doesn't even exist. Still, LLMs appear to consistently follow the values of secular\rational people who strive for self-expression (sounds like me😅) To show it they collect and release 200K human-model chats+feedback, 5 languages and 21 LLMs 🤖📈🧠

Pressing matters in evaluation from GEMs talk Remember, exciting questions drive science, exciting answers follow. Setting the right goal may make all their sota chasing worthwhile. Make an insightful dataset, lead by evaluation 🤖📈🧠

The talk for our work on Retrospective Learning from Interactions, which will be in ACL (once I figure out how to squeeze it shorter) Gist: autonomous post-training from conversational signals for LLM bootstrapping ... look ma, no annotations! no hand-holding! 🙌📈🚀 www.youtube.com/watch?v=qW8S...

ICML proposes we use sophisticated jailbreaks in our papers? Ones that trick the reviewers but do not raise our scores? Proposals?

Can LLMs learn social skills by playing games? A blogpost on human-model interaction, games, training and testing LLMs research.ibm.com/blog/LLM-soc... 🤖📈🧠

We don't understand loss spikes, that's clear. We've learned recently that data deterministically makes spikes regardless of optimizer. What did we see when we stopped pretraining, and then continue? A huge spike and never a recovery? Why? Apparently the momentum matters, a lot. 🤖📈🧠

Eu calls for evaluator experts to counsel about how to evaluate models (can't belong to a training company, smart) digital digital-strategy.ec.europa.eu/en/news/comm... #safety #eticalAI #AI 🤖📈🧠 Tag people who should know or should pass the knowledge

The ML\LLM AI world is overcrowded✅ and it's hard to find majestic new projects❌ Here is (updating) list of thoughts, premature ideas, and research directions that I cannot pursue alone, but I believe can make radical changes Please fight me, discuss, and ask Disrrruption, aye?🏴‍☠️ 🤖📈🧠

Surprisingly good tokenization workshop, resurfaced thoughts:🧠📈🤖 Why isn't tokenization learned? Can we do an evolutionary algorithm, or train a tokenization scheme on the pretrained or meta learn on something fast-pretrain (loss at the beginning of training?) Let's discuss👇

Do models really understand humans (ToM)? Can your model do it? Beat them at the neurips Competition. 🤖📈🧠

How do you test a new model? What's super challenging for models? The @kaggle.com team are starting a new way to crowd evaluate models, create your own benchmarks.🚀 (i'll tell you mine🧵 if you tell me yours) 🤖📈🧠 kaggle-task-generator-564531943159.us-west1.run.app

Models self-teach slightly more complex skills than the teacher knows And they can do so again and again until it is not just slightly. My questions:🧵

Teaching ChatGPT to tell everyone how handsome I am

Imagine a hacker causing ChatGPT to genuinely believe the stock market is crashing. No breach, no code, all an attacker needs is a malicious prompt and clicking 👍to change the model's weights. Our research paper demonstrates a vulnerability in LLMs RLHF mechanism: arxiv.org/abs/2507.02850 📈🤖🧠

The Swedish National Archives shared 1.2m images of handwritten documents from 1600-1900+ In addition, they share a model for writing to text Such marvelous data opportunities, kudos 📈🤖🧠 medium.com/@linnea.karl... Example: sok.riksarkivet.se/om-soktjanst... huggingface.co/datasets/Rik...

For the past ~2 months we have been working on training reasoning models on TextArena.ai games. Soon, a paper introducing what we think is a promising new paradigm (+ scaling and analysis). However, to get more feedback on the structure and implementation, we want to open-source the code now.

"Over four months, LLM users consistently underperformed at neural, linguistic, and behavioral levels. These results raise concerns about the long-term educational implications of LLM reliance and underscore the need for deeper inquiry into AI's role in learning." arxiv.org/abs/2506.08872

🚀 Technical practitioners & grads — join to build an LLM evaluation hub! Infra Goals: 🔧 Share evaluation outputs & params 📊 Query results across experiments Perfect for 🧰 hands-on folks ready to build tools the whole community can use Join the EvalEval Coalition here 👇 forms.gle/6fEmrqJkxidy...

When Claude, Gemini, o3 battle each other for world domination in Diplomacy: DeepSeek turned warmongering tyrant. Claude wouldn't lie—everyone exploited it ruthlessly. o3 orchestrated a secret coalition, backstabbed every ally, and won. tx textArena.ai and friends every.to/p/diplomacy 🧠🤖 #AI #LLM

Remember Toyota scandal, where cars just "better behaved" when tested? If someone tested you, wouldn't you? And if you tested LLMs? Finding: LLMs can tell when they are evaluated🧠 We can only wait to see how they act on it alphaxiv.org/pdf/2505.23836 📈🧠🤖

Gradient norms rise at the end of training, and a new theory connects that to the learning loss when using weight decay. AdamC and SGDC(orrected) multiply the weight decay by the learning rate and fix it. alphaxiv.org/pdf/2506.02285 📈🤖🧠

Want people to interact with you? 💬 Open your DMs! Go to your Chats tab → tap the settings icon ⚙️ (Don’t go to Settings directly — that won’t work 😅)

They don't only "not consistently boost" they underperform :-(

Training LLMs on many games can generalize and teaches reasoning that holds in new environments. However, weighting the games is complicated, so merging (my beloved fusing in the title) is used. 📈🤖🧠 alphaxiv.org/pdf/2505.16401

I am glad that you found our paper entertaining! This is a great point for my follow-up thread on the implications of concept incongruence. Our main goal is to raise awareness and provide clarity around concept incongruence.

On the one hand, concept incongruence is very common. In role-play, “death” is our most vivid test case, yet the same tension appears any time a model must juggle incompatible constraints. This also shows up in political discussions, alignment (helpful vs. harmless), and essentially everywhere.

Highly entertaining paper and writeup, but does it really matter? Is it important that models can't abstain on counterfactuals? Or that the leak information?

I don't fully 100% agree/conform with this, but it's a good list!

Don't play games, create them

The cognitive side of babyLMs is up for the reading

Did you remember all the things to do before sharing a figure? Remove "_" from the labels save as pdf ... A lot of deeper tips to writing are in the shared document, but somehow this practical list was always the most popular. Good luck in #neurips docs.google.com/document/d/1... 🧠🤖📈 #LLM #AI

Has anyone written anything about *scraping and text processing* for internet pretraining data? Practical details, which tools are used, which webpage elements are considered, how HTML to text conversion is done? (I know about work on quality filters, relevant but not quite what I'm looking for)

Close your books, test time! The evaluation pipelines are out, baselines are released & the challenge is on There is still time to join and We are excited to learn from you on pretraining and human-model gaps *Don't forget to fastEval on checkpoints github.com/babylm/evalu... 📈🤖🧠 #AI #LLMS

Base Models Beat Aligned Models at Randomness and Creativity. Peter West & @cgpotts.bsky.social tell us that alignment doesn't only extract abilities hidden in the pretraining, it also hides other abilities: 📈🤖🧠 #LLM #AI #creativity

So far the best experience in getting knowledge out of bluesky! Highly recommended bsky.app/profile/did:... T.h @nsaphra.bsky.social Feilling with goods @tedunderwood.me