What if the way we verify synthetic code is limiting model performance? In our latest work we uncover the Verification Ceiling Problem: strict “all tests must pass” rules throw away useful data, while weak tests let errors through.

We’re not your average lab. We’re a hybrid research environment dedicated to revolutionizing the ML space. And we’re hiring a Senior Research Scientist to co-create with us. If you believe in research as a shared, global effort — this is your chance.

I'm excited to share that I'll be stepping into the role of Head of @cohereforai.bsky.social. It's an honor and a responsibility to lead such an extraordinary group of researchers pushing the boundaries of AI research.

While effective for chess♟️, Elo ratings struggle with LLM evaluation due to volatility and transitivity issues. New post in collaboration with AI Singapore explores why Elo falls short for AI leaderboards and how we can do better.

Breaking into AI research is harder than ever, and early-career researchers face fewer chances to get started. Entry points matter. We started the Scholars Program 3 years ago to give new researchers a real shot — excited to open applications for year 4✨

🌍 Language shapes how we think and connect—but most AI models still struggle beyond English. @microsoft.com's July seminar discussed how we can bridge the gap and build #AIforEveryone with @mziizm.bsky.social of @cohere.com. 📽️ www.microsoft.com/en-us/resear...

ACL day 2 ✨

🖼️ Most text-to-image models only really work in English. This limits who can use them and whose imagination they reflect. We asked: can we build a small, efficient model that understands prompts in multiple languages natively?

Everyone talks about GEB (I agree, it's a gem) but Hofstadter's Analogy book is criminally underrated. If you're working on learning intelligence through language understanding, it’s a must-read.

🍋 Squeezing the most of few samples - check out our LLMonade recipe for few-sample test-time scaling in multitask environments. Turns out that standard methods miss out on gains on non-English languages. We propose more robust alternatives. Very proud of this work that our scholar Ammar led! 🚀

London has me under its spell. every. single. visit.

🚨LLM safety research needs to be at least as multilingual as our models. What's the current stage and how to progress from here? This work led by @yongzx.bsky.social has answers! 👇

Over 7000 languages are spoken worldwide 🌐, but AI safety efforts focus on only a fraction of them. Our latest paper draws on our multi-year efforts with the wider research community to explore why this matters and how we can bridge the AI language gap.

📢 The Copenhagen NLP Symposium on June 20th! - Invited talks by @loubnabnl.hf.co (HF) @mziizm.bsky.social (Cohere) @najoung.bsky.social (BU) @kylelo.bsky.social (AI2) Yohei Oseki (UTokyo) - Exciting posters by other participants Register to attend and/or present your poster at cphnlp.github.io /1

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

Goodhart's law rules everything around me.

1/ Science is only as strong as the benchmarks it relies on. So how fair—and scientifically rigorous—is today’s most widely used evaluation benchmark? We took a deep dive into Chatbot Arena to find out. 🧵

Not in Singapore for #ICLR2025 but our lab’s work is! In particular, I am very proud of these collaborations: ✨INCLUDE (spotlight) — models fail to grasp regional nuances across languages 💎To Code or Not to Code (poster) — code is key for generalizing beyond coding tasks

🚨 Excited to share our latest paper! Multilingual LLMs are getting really good. But the way we evaluate them? Not the best sometimes. 🌟 We show how decades of lessons from Machine Translation can help us fix it

Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative! 🌍 18 languages (high-, mid-, low-) 📚 21k questions (55% require image understanding) 🧪 STEM, social science, reasoning, and practical skills

Big news from WMT! 🎉 We are expanding beyond MT and launching a new multilingual instruction shared task. Our goal is to foster truly multilingual LLM evaluation and best practices in automatic and human evaluation. Join us and build the winning multilingual system! www2.statmt.org/wmt25/multil...

☀️ Summer internship at Cohere! Are you excited about multilingual evaluation, human judgment, or meta-eval? Come help us explore how a rigorous eval really looks like while questioning the status quo in LLM evaluation. I’m looking for an intern (EU timezone preferred), are you interested? Ping me!

Command🅰️ technical report is out. Information-dense. Detailed. Pretty. Simply A+! 💎: cohere.com/research/pap...

Good morning Paris

✨👓 Aya Vision is here 👓✨ A multilingual, multimodal model designed to understand across languages and modalities (text, images, etc) to bridge the language gap and empower global users!

Excited to share insights from our new paper on evaluating LLMs in multi-session coding interactions! 📚📚📚 We introduce MEMORYCODE, a novel dataset to assess how well LLMs track & execute coding instructions across multiple sessions, mimicking real-world collaboration.

INCLUDE is going to ICLR! Huge congrats to all my co-authors!

This #Neurips2024 was the perfect way to end this year. So long Vancouver!

Day 2 #neurips2024, let the fun officially begin. On a separate note, as much as I love Amsterdam I'm mountain-deprived and only have eyes for this glorious view this week.

Good morning Vancouver! It's a lovely day to see old friends and make new ones.

Aya Expanse technical report is here and I couldn’t be prouder! 🎉 Check out the report: arxiv.org/abs/2412.04261

🚀 Our mission to strengthen the multilingual open-source ecosystem continues!👇

AI amplifying biorisk has been a major topic in policy & governance work. But does the available evidence match this level of attention? 🦠 ⚠️ Our new paper looks at the science underpinning ideas that AI could increase biorisks. arxiv.org/abs/2412.01946

see you in a few hours 🤩

Good performance shouldn’t mean 'just in English' anymore 🪩 We provide a robust way to assess models with a new benchmark that captures in-language nuances and cultural contexts.

I'm always thinking about the aspects of research that aren't discussed enough online, and one key area is effective scientific communication. Very excited about this panel discussion featuring two amazing researchers: @jayalammar.bsky.social and @shaynelongpre.bsky.social

Laura is amazing for seeing this project through! Check out her paper on studying the reasoning capabilities of LLMs and influences from the training data 🔮

does anyone else mutter "banksy" in their head when they type "bsky" instead of "bluesky"?

It's a beautiful day to immerse in overleaf drafts

hello world! #NLProc