Evaluating language models is tricky, how do we know if our results are real, or due to random chance? We find an answer with two simple metrics: signal, a benchmark’s ability to separate models, and noise, a benchmark’s random variability between training steps 🧵

📢 New #COLM2025 paper 📢 Standard benchmarks give every LLM the same questions. This is like testing 5th graders and college seniors with *one* exam! 🥴 Meet Fluid Benchmarking, a capability-adaptive eval method delivering lower variance, higher validity, and reduced cost. 🧵

🚀 Introducing Fluid Benchmarking—an adaptive way to evaluate LLMs. Inspired by psychometrics, it tailors which questions to ask based on each model’s capability, making evals more efficient & reliable. 🧵

🔭 Science relies on shared artifacts collected for the common good. 🛰 So we asked: what's missing in open language modeling? 🪐 DataDecide 🌌 charts the cosmos of pretraining—across scales and corpora—at a resolution beyond any public suite of models that has come before.

Today we're unveiling OLMoTrace, a tool that enables everyone to understand the outputs of LLMs by connecting to their training data. We do this on unprecedented scale and in real time: finding matching text between model outputs and 4 trillion training tokens within seconds. ✨

🚨I too am on the job market‼️🤯 I'm searching for faculty positions/postdocs in multilingual/multicultural NLP, vision+language models, and eval for genAI! I'll be at #NeurIPS2024 presenting our work on meta-evaluation for text-to-image faithfulness! Let's chat there! Papers in🧵, see more: saxon.me

the science of LMs should be fully open✨ today @akshitab.bsky.social @natolambert.bsky.social and I are giving our #neurips2024 tutorial on language model development. everything from data, training, adaptation. published or not, no secrets 🫡 tues, 12/10, 9:30am PT ☕️ neurips.cc/virtual/2024...

Excited to present MediQ at #NeurIPS ! 📍Stop by my poster: East Exhibit Hall A-C #4805📷 🕚Thu, Dec 12 | 11am–2pm 🗓️tinyurl.com/mediq2024 Love to chat about anything--reasoning, synthetic data, multi-agent interaction, multilingual nlp! Message me if you want to chat☕️🍵🧋

Want to predict the task performance of LMs before pretraining them? We develop task scaling laws and model ladders, which predict the accuracy on individual tasks by OLMo 2 7B & 13B models within 2 points of absolute error. The cost is 1% of the compute used to pretrain them.

Excited to be at #NeurIPS next week in 🇨🇦! Please reach out if you want to chat about LM post-training (Tülu!), data curation, or anything else :) I'll be around all week, with two papers you should go check out (see image or next tweet):

Touching down in Vancouver 🛬 for #NeurIPS2024! I'll be presenting our "Consent in Crisis" work on the 11th: arxiv.org/abs/2407.14933 Reach out to catch up or chat about: - Training data / methods - AI uses & impacts - Multilingual scaling

Come chat with me at #NeurIPS2024 and learn about how to use Paloma to evaluate perplexity over hundreds of domains! ✨We have stickers too✨

Building/customizing your own LLM? You'll want to curate training data for it, but how do you know what makes the data good? You can try out recipes👩‍🍳 iterate on ✨vibes✨ but we can't actually test all possible combos of tweaks,,, right?? 🙅‍♂️WRONG! arxiv.org/abs/2410.15661 (1/n) 🧵

LMs are used to process text from many topics, styles, dialects, etc., but how well do they do? 📈 Evaluating perplexity on just one corpus like C4 doesn't tell the whole story 📉 ✨📃✨ We introduce Paloma, a benchmark of 585 domains from NY Times to r/depression on Reddit.