How well do LLMs handle multilinguality? 🌍🤖 🔬We brought the rigor from Machine Translation evaluation to multilingual LLM benchmarking and organized the WMT25 Multilingual Instruction Shared Task spanning 30 languages and 5 subtasks.

River, Yinhong and I will all be in person and we look forward to the discussions!

We’re thrilled to announce that some of our research will be presented at @emnlpmeeting.bsky.social next week! 🥳 If you’re attending the conference, don’t miss the chance to explore our work and connect with our team.

“Individually, we are one drop. Together, we are an ocean.” - Ryunosuke Satoro ✨ Cohere Labs is excited to announce Connect - a 3-day virtual conference celebrating the power of collaboration in open science!

🌍Most multilingual instruction data starts as English and translation can’t capture cultural nuance or linguistic richness What if we optimized prompts instead of completions? That’s the focus of our most recent work on prompt space optimization for multilingual synthetic data🗣️

🚀 Global MMLU Lite is now live on Kaggle Benchmarks! Developed by @cohereforai.bsky.social, it spans 16 languages with both Culturally Sensitive & Agnostic samples - helping researchers uncover cultural & linguistic biases in multilingual evaluation.

Global AI deserves reproducible and transparent evaluation. 🌎 With Global MMLU Lite now part of @kaggle.com Benchmarks, you can track the multilingual performance of top models as well as test your own! Check out the leaderboard and notebook linked below.

This month, we've been very excited to welcome Joelle Pineau, @cohere.com's new Chief AI Officer. We look forward to working together on frontier research - advancing the science of building models that are robust, capable, and impactful in the real world.

Keynote talk: Optimizing Multilinguality Post Training. Can multilingual ability be boosted at post training? Julia Kreutzer from @cohereforai.bsky.social explores RL, test-time scaling & data distillation to improve open-ended tasks across languages. 🌍✨ #MELTWorkshop2025

Let's do the venue justice. Very excited for today's multilingual workshops at #COLM2025 💙

Today at COLM, Cohere Labs Sr Research Scientist, @juliakreutzer.bsky.social will be presenting at 2 workshops. First, the Multilingual Data Quality Signals workshop, bringing together researchers across disciplines to discuss & present research on data quality signals in multilingual data.

Today at COLM, we are excited to share our work Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation, during Poster Session 4, 4:30 - 6:30pm. Come connect with paper authors @juliakreutzer.bsky.social and @kocmitom.bsky.social.

💡A collaborative➕diverse team is key. In real life as in the LLM world 💪🦾 Check out our latest work that builds on this insight. 👇

Is Best-of-N really the best use of your inference compute? Introducing Fusion-of-N: a simple and powerful way to advance inference and distillation beyond Best-of-N.

We’re not your average lab. We’re a hybrid research environment dedicated to revolutionizing the ML space. And we’re hiring a Senior Research Scientist to co-create with us. If you believe in research as a shared, global effort — this is your chance.

What if the way we verify synthetic code is limiting model performance? In our latest work we uncover the Verification Ceiling Problem: strict “all tests must pass” rules throw away useful data, while weak tests let errors through.

I'm excited to share that I'll be stepping into the role of Head of @cohereforai.bsky.social. It's an honor and a responsibility to lead such an extraordinary group of researchers pushing the boundaries of AI research.

Papers In The Park 14. Last one of the season! Still great weather. Surprising. Anthony is presenting the “Why Language Models Hallucinate”. Thanks to @cohereforai.bsky.social for the copies and pizza.

🚨 Rare opportunity: Cohere Labs is hiring a Research Scientist! If you’re passionate about studying fundamental AI problems and working in a globally collaborative, open-science environment, this is for you. Apply here: jobs.ashbyhq.com/cohere/7ec9e...

It’s papers in the park 7! Thanks to @cohereforai.bsky.social for the papers and the pizza, and to Alvin and Anthony for organizing. It’s easily one of funnest paper reads in the city!

Breaking into AI research is harder than ever, and early-career researchers face fewer chances to get started. Entry points matter. We started the Scholars Program 3 years ago to give new researchers a real shot — excited to open applications for year 4✨

While effective for chess♟️, Elo ratings struggle with LLM evaluation due to volatility and transitivity issues. New post in collaboration with AI Singapore explores why Elo falls short for AI leaderboards and how we can do better.

Applications are now open for the next cohort of the Cohere Labs Scholars Program! 🌟 This is your chance to collaborate with some of the brightest minds in AI & chart new courses in ML research. Let's change the spaces breakthroughs happen. Apply by Aug 29.

Can we improve the performance of LLMs during inference without the need for extensive sampling OR special reward models? 🤔 Our latest work introduces a new inference time scaling recipe that is sample-efficient, multilingual, and suitable for multi-task requirements. 🍋

It’s been two years since cross-lingual jailbreaks were first discovered. How far has the multilingual LLM safety research field advanced? 🤔 📏 Our comprehensive survey reveals that there is still a long way to go.

Over 7000 languages are spoken worldwide 🌐, but AI safety efforts focus on only a fraction of them. Our latest paper draws on our multi-year efforts with the wider research community to explore why this matters and how we can bridge the AI language gap.

🚀🌍The rapid advancement of multilingual large language models (mLLMs) is exciting, but are we evaluating them effectively? Our new paper explores how we can improve generative evaluations for mLLMs by learning from machine translation (MT) evaluation practices. 🔎

🚀 We are excited to introduce Kaleidoscope, the largest culturally-authentic exam benchmark. 📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation

We’re excited to bring back Expedition Aya 🌍 A 6-week open build challenge to accelerate ML research progress in multilingual, multimodal and efficiency. Join us to expand the world that AI sees.

Just 2 days after launch, Aya Vision is trending on @hf.co 🔥🔥 We launched open-weights with the goal of making VLM breakthroughs accessible to the research community - so exciting to see such a positive response. huggingface.co/CohereForAI/...

We're particularly proud to release Aya Vision 8B - it's compact 🐭 and efficient 🐎, outperforming models up to 11x its size 📈. Releasing open weights helps to make breakthroughs in VLMs accessible to the research community.

Who will triumph - Aya Vision or its creators? 🤼 We challenged the research team behind Aya Vision to mystery object trivia. 💎🌽🧈 www.youtube.com/watch?v=iQEd...

Introducing ✨ Aya Vision ✨ - an open-weights model to connect our world through language and vision Aya Vision adds breakthrough multimodal capabilities to our state-of-the-art multilingual 8B and 32B models. 🌿

👀

The C4AI Research Grant program is proud to have supported a project focused on building LLM tools for teachers 🧑‍🏫 This project focused on adapting educational materials to students’ skill levels, ensuring more effective and responsible AI integration in classrooms.

Merci, Paris! 🇫🇷It’s been an incredible week at the AI Action Summit, discussing our work on AI sustainability, technology that serves a multilingual world and AI safety. 🌿 🌎 🦺 We are honoured to represent at the summit.

An important topic in AI is the climate impacts of the energy-intensive computing hardware needed to train and deploy AI models ⚡ Our latest policy primer explores ways to move towards more sustainable AI. 🌱

Have you joined our open science community? Our community is a space where researchers, engineers, linguists, social scientists and lifelong learners connect and collaborate with each other, from all over the world 🌏 Join now: https://share.hsforms.com/10OrjljwpQ52ILJA6ftENIwch5vw

Introducing Fairness of Deep Ensembles. 🎉 In our latest cross-institutional collaboration, we explore the possibility of achieving greater fairness by using an imbalanced dataset instead of a balanced one, and how the use of ensembles could further amplify this impact.

Have you checked out our Community Talks Youtube Playlist? 📹 We have over 100 guest speaker sessions on a variety of topics including LLMs, Computer Vision, Reinforcement Learning, NLP and more! Our most recent sessions featured presentations from Andrei Panferov, Jillian Fisher and Yunsu Kim.

"Anyone who is serious about what the next generation of models is knows it can't be the current" Thanks to @baratunde.com for hosting Head of Cohere For AI, @sarahooker.bsky.social on the latest episode of Life with Machines. Check out their full conversation on YouTube: youtu.be/-BsobAoOJvk

Our paper is accepted to ICLR! INCLUDE: Evaluating Multilingual LLMs with Regional Knowledge (arxiv.org/abs/2411.19799) A benchmark of ~200k QA pairs across 44 languages, capturing real-world cultural nuances. A collaborative effort led by @cohereforai.bsky.social, with contributors worldwide. /1

Are you a researcher working towards a peer-reviewed paper or other research output (e.g., datasets, code), and you’d like to make use of Cohere’s state-of-art models?

Aya Expanse, our open-weight 32B model, outperforms drastically larger models including Claude, Mistral Large 2, & Llama 405B on Scale's Private Multilingual Protocol. We are proud to work on global AI that is efficient and accessible 🔥

On Scale AI's private multilingual protocol, Aya Expanse is indexed as the best open-weights model. Additionally, in some languages we're outperforming: 🔒proprietary models 🐘larger models ⛰️models built by more researchers with more infrastructure Lots to be proud of today.

📣 Come share your work with our open science community! Have you recently published a paper? Are you a student looking to present your recent findings? Are you a seasoned professional in the field of AI and looking to share some knowledge and expertise?

We are committed to making meaningful progress in machine learning research through open collaboration. Follow this 🧵to stay on top of our research contributions.

Hello, world. ✨ Cohere For AI is @cohere.com's research lab and open science initiative that seeks to solve complex machine learning problems. Join us in exploring the unknown, together. cohere.com/research