📢 New #COLM2025 paper 📢 Standard benchmarks give every LLM the same questions. This is like testing 5th graders and college seniors with *one* exam! 🥴 Meet Fluid Benchmarking, a capability-adaptive eval method delivering lower variance, higher validity, and reduced cost. 🧵

🚀 Introducing Fluid Benchmarking—an adaptive way to evaluate LLMs. Inspired by psychometrics, it tailors which questions to ask based on each model’s capability, making evals more efficient & reliable. 🧵

📓 New from Ai2: we’ve released source code that shows how we built AskOlmo, our Discord chatbot powered by our Olmo model family. It’s a peek behind the curtain—so you can see how it all came together. 👇

🚀 New in the Ai2 Playground: side-by-side comparison is live. Compare two Ai2 models with the same prompt and see the results next to each other. ⚖️🆚

🌍☀️❄️ Can AI forecast year-to-year differences in the seasons? New research with @metoffice.gov.uk shows our ACE2 ML model demonstrates seasonal forecasting skill—matching traditional physics-based methods while using dramatically less compute. 🧵

✨Meet OLMoASR✨ By pairing our curated 1M-hour dataset with a powerful architecture, we've built open ASR models that achieve competitive performance with models like Whisper. We're open-sourcing data, code and models to help the community build more robust and transparent ASR.

🎙️ Say hello to OLMoASR—our fully open, from-scratch speech-to-text (STT) model. Trained on a curated audio-text set, it boosts zero-shot ASR and now powers STT in the Ai2 Playground. 👇

We're honored to be named a @fastcompany.com Corporate Wellness Innovator 2025 (Tech)! 🏆 Our Seattle HQ was built for well-being with fresh air, outdoor spaces, and movement-friendly designs to help our team thrive.

Today we’re releasing agent-baselines, a suite of 22 classes of AI agents for science—including 9 open-source research-tuned agents like our state-of-the-art, benchmark-leading Asta v0. 🚀🔬 Part of our Asta ecosystem to advance scientific AI. 👇

As part of Asta, our initiative to accelerate science with trustworthy AI agents, we built AstaBench—the first comprehensive benchmark to compare them. ⚖️

Introducing Asta—our bold initiative to accelerate science with trustworthy, capable agents, benchmarks, & developer resources that bring clarity to the landscape of scientific AI + agents. 🧵

🚨 SciArena update + evaluation of new models including GPT-5! 🚨 With thousands of new votes, new LLMs are reshaping our leaderboard for scientific literature tasks. o3 still leads—but GPT-5, Claude Opus 4.1, & more are closing the gap.

🚀 In March, we launched Paper Finder, an LLM-powered literature search agent that surfaces papers other tools miss. Now, we’re releasing an open-source snapshot to enable others to inspect & build on it—and reproduce the results. 🧵

Evaluating language models is tricky, how do we know if our results are real, or due to random chance? We find an answer with two simple metrics: signal, a benchmark’s ability to separate models, and noise, a benchmark’s random variability between training steps 🧵

📢 New paper from Ai2: Signal & Noise asks a simple question—can language model benchmarks detect a true difference in model performance? 🧵

We’re releasing early pre-training checkpoints for OLMo-2-1B to help study how LLM capabilities emerge. They’re fine-grained snapshots intended for analysis, reproduction, and comparison. 🧵

LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇

With fresh support of $75M from NSF and $77M from NVIDIA, we’re set to scale our open model ecosystem, bolster the infrastructure behind it, and fast‑track reproducible AI research to unlock the next wave of scientific discovery. 💡

🤖✨ What if models that take action in the physical world could think through your instructions? Meet MolmoAct, our new fully open Action Reasoning Model (ARM) that does just that. 🧵

🤖 💬 You can now chat with OLMo 2 32B Instruct, our most capable language model, directly in our Discord by tagging @AskOLMo!

We’re participating in this year’s Generative Red Teaming Challenge (GRT 3) at #defcon in Las Vegas 🛡️ Starting Thursday, attendees will stress-test LLM evals through live public red-teaming—helping advance the state of AI evaluations. 🧵

🛰️ 🔭 Say hello to Galileo, an open-source multimodal model designed to process many kinds of earth observations at once—driving applications such as mapping agricultural land, detecting floods, and monitoring marine pollution.

📝 olmOCR v0.2.1 has arrived with new models! Our open‑source OCR engine now reads tougher docs with greater precision—and it’s still 100 % open. 👇

New interview drop! ⚠️ Ai2 Senior Research Scientist @natolambert.bsky.social sat down with @kylelwiggers.bsky.social, Ai2’s new Comms Lead, to talk about what sets Ai2 apart from other frontier AI labs, and our role in the research community. Watch the highlights. 👇

🌎🛡️ On #WorldRangerDay, we’re proud to share that Honduras is expanding the use of @EarthRangerTech – our real‑time wildlife‑protection tech platform – to advance zero‑deforestation and safeguard biodiversity. 🧵

Ai2 is excited to be at #ACL2025 in Vienna, Austria this week. Come say hello, meet the team, and chat about the future of NLP. See you there! 🤝📚

🚀 Ai2’s official subreddit has arrived. Discuss our latest work and happenings in the AI community, from LLMs to bleeding-edge scientific work. We look forward to seeing you there. 👋

issues w preference LM benchmarks: 🐡data contains cases where the "bad" response is just as good as chosen one 🐟model rankings can feel off (claude ranks lower than expected) led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025

In our new paper, “Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries,” we find that adding just a bit of missing context can reorder model leaderboards—and surface hidden biases. 🧵👇

Great science starts with great questions. 🤔✨ Meet AutoDS—an AI that doesn’t just hunt for answers, it decides which questions are worth asking. 🧵

A new model enters SciArena. 👀 Welcome Moonshot AI's Kimi K2! SciArena lets you benchmark models across scientific literature tasks, applying a crowdsourced LLM evaluation approach to the scientific domain. 🧪 Learn more and try SciArena here: buff.ly/J2kYM0Q

We’ve upgraded ScholarQA, our agent that helps researchers conduct literature reviews efficiently by providing detailed answers. Now, when ScholarQA cites a source, it won’t just tell you which paper it came from–you’ll see the exact quote, highlighted in the original PDF. 🧵

Ai2’s work spans robotics. We lead cutting-edge research to develop the next generation of intelligent robots, safely trained in advanced simulation environments to automate routine tasks and improve daily life. 🧵

This week is #ICML in Vancouver, and a number of our researchers are participating. Here's the full list of Ai2's conference engagements—we look forward to connecting with fellow attendees. 👋

We've added Grok 4, the latest model from xAI, to our SciArena platform! SciArena allows you to benchmark models across scientific literature tasks, applying a crowdsourced LLM evaluation approach to the scientific domain. 👇

Did you know? Over 170 partners across more than 60 countries rely on Skylight, our platform to help visualize the movement of vessels, to guide decisions and protect what matters most—from human lives to coastal communities. 🧵

Training powerful models depends on diverse, high-quality data. But while there’s growing interest in applying AI across all sectors, data remains a bottleneck. To tackle this challenge, we developed FlexOlmo. 🧵

A novel approach from the Allen Institute for AI enables data to be removed from an artificial intelligence model even after it has already been used for training. www.wired.com/story/flexol...

Introducing FlexOlmo, a new paradigm for language model training that enables the co-development of AI through data collaboration. 🧵

OLMo 1B, OLMo 7B, OLMo 13B, Tülu 8B, and Tülu 70B are back in the Ai2 Playground! The Cirrascale API platform is now hosting several open models on the Ai2 Playground: Our OLMo and Molmo models, as well as our open-weight Tülu models.

It’s now easier than ever to deploy, fine-tune, and scale our powerful open-source AI models via API.🧵

Introducing IFBench, a benchmark to measure how well AI models follow new, challenging, and diverse verifiable instructions. Top models like Gemini 2.5 Pro or Claude 4 Sonnet are only able to score up to 50%, presenting an open frontier for post-training. 🧵

AI practitioners need to know if their models are performant, safe, biased, or hallucinating. That’s why Ai2 offers a range of evaluation frameworks and benchmarks, all of which are open and accessible. 🧵

Today we released SciArena, an open evaluation platform where researchers can compare and vote on foundation models for scientific literature tasks. 👇

Introducing SciArena, a platform for benchmarking models across scientific literature tasks. Inspired by Chatbot Arena, SciArena applies a crowdsourced LLM evaluation approach to the scientific domain. 🧵

How can we better understand how models make predictions and which components of a training dataset are shaping their behaviors? In April we introduced OLMoTrace, a feature that lets you trace the outputs of language models back to their full training data in real time. 🧵

🚨 We're hiring a #ResearchScientist in #AI for Scientific Discovery at Ai2! Are you passionate about intelligent agents, data-driven discovery, and AI systems that accelerate science? Join us in shaping the future of research. 🧬🧠 Apply now: job-boards.greenhouse.io/thealleninst...

Today we’re releasing a prototype of Genesys, an autonomous multi-agent LLM discovery system that aims to discover new types of language model architectures. We found Genesys can discover novel architectures competitive with the industry-standard transformer. 🧵

Hear Ai2 Senior Researcher @natolambert speak at @VentureBeat Transform Wednesday, 6/25! Join us at 2:30 PM PT for his talk, “A Taxonomy for Next-Generation Reasoning Models.” See you there! #VBTransform

How well can today’s models generalize, compose, or even innovate on unseen problems? OMEGA Ω is a new math benchmark that pushes LLMs beyond pattern-matching to test true mathematical reasoning. ⚡ allenai.org/blog/omega