When reading AI reasoning text (aka CoT), we (humans) form a narrative about the underlying computation process, which we take as a transparent explanation of model behavior. But what if our narratives are wrong? We measure that and find it usually is. Now on arXiv: arxiv.org/abs/2508.16599

📊 Preliminary ranking of WMT 2025 General Machine Translation benchmark is here! But don't draw conclusions just yet - automatic metrics are biased for techniques like metric as a reward model or MBR. The official human ranking will be part of General MT findings at WMT. arxiv.org/abs/2508.14909

Happy to see this work accepted to #EMNLP2025! 🎉🎉🎉

✨We are thrilled to announce that over 3200 papers have been accepted to #EMNLP2025 ✨ This includes over 1800 main conference papers and over 1400 papers in findings! Congratulations to all authors!! 🎉🎉🎉

The Echoes in AI paper showed quite the opposite with also a story continuation setup. Additionally, we present evidence that both *syntactic* and *discourse* diversity measures show strong homogenization that lexical and cosine used in this paper do not capture.

GPT-5 lands first place on NoCha, our long-context book understanding benchmark. That said, this is a tiny improvement (~1%) over o1-preview, which was released almost one year ago. Have long-context models hit a wall? Accuracy of human readers is >97%... Long way to go!

🗓️29 July, 4 PM: Automated main concept generation for narrative discourse assessment in aphasia. w/ @marisahudspeth.bsky.social, Polly Stokes, Jacquie Kurland, and @brenocon.bsky.social 📍Hall 4/5. Come by to chat about argumentation, narrative texts, policy & law, and beyond! #ACL2025NLP

Excited to present two papers at #ACL2025! 🗓️30 July, 11 AM: 𝛿-Stance: A Large-Scale Real World Dataset of Stances in Legal Argumentation. w/ Douglas Rice and @brenocon.bsky.social 📍At Hall 4/5. 🧵👇

📣 Life update: Thrilled to announce that I’ll be starting as faculty at the Max Planck Institute for Software Systems this Fall! I’ll be recruiting PhD students in the upcoming cycle, as well as research interns throughout the year: lasharavichander.github.io/contact.html

For EMNLP 2025’s special theme of "Advancing our Reach: Interdisciplinary Recontextualization of NLP", we are organizing a panel of experts, and would like input from the community at large as we prepare. Please take a moment to fill in this survey: forms.office.com/r/pWFFA0Gss1

A new definition for AGI just dropped, and it is a bad one.

Now accepted to #COLM2025 @colmweb.org 🇨🇦🎉

What should Machine Translation research look like in the age of multilingual LLMs? Here’s one answer from researchers across NLP/MT, Translation Studies, and HCI. "An Interdisciplinary Approach to Human-Centered Machine Translation" arxiv.org/abs/2506.13468

Extremely interesting new task that gives a model a literary text, plus a critical essay about it — with one quotation masked. Can the model figure out which quotation from the original work would support these claims? Best-performing models exceed human readers. #MLSky arxiv.org/abs/2506.030...

Tired of AI slop? Our work on "Frankentexts" shows how LLMs can stitch together random fragments of human writing into coherent, relevant responses to arbitrary prompts. Frankentexts are weirdly creative, and they also pose problems for AI detectors: are they AI? human? More 👇

🤔 What if you gave an LLM thousands of random human-written paragraphs and told it to write something new -- while copying 90% of its output from those texts? 🧟 You get what we call a Frankentext! 💡 Frankentexts are surprisingly coherent and tough for AI detectors to flag.

Interested in crosslingual memorization? Check out our new work :) Congrats to Emir, Alisha, and Minh for putting together their first research paper 🎉

LLMs memorize novels 📚 in English. But what about existing translations? Or translations into new languages? Our 🦉OWL dataset (31K/10 languages) shows GPT4o recognizes books: 92% English 83% official translations 69% unseen translations 75% as audio (EN)

That's just insane! @naaclmeeting.bsky.social Link to original report: www.lemonde.fr/en/internati...

We have also added #Maverick which is arguably better but (similarly to other models) struggles when the evidence is not easy to retrieve (ie requires more reasoning than simple retrieval -- hard set). More here: novelchallenge.github.io

Llama 4's massive context window is impressive! However, the best Llama model for long-context understanding over books is still Llama 3.1 405B. Llama 4 Scout is especially bad at our NoCha benchmark, performing below random chance.

We have updated #nocha (long-context benchmark measuring how well models process book-length narratives) with #Llama4 Scout. Sadly, the performance was below the random level, much lower than the reported model's performance on a retrieval task (needle in the haystack). novelchallenge.github.io

We have updated #nocha, a leaderboard for reasoning over long-context narratives 📖, with some new models including #Gemini 2.5 Pro which shows massive improvements over the previous version! Congrats to #Gemini team 🪄 🧙 Check 🔗 novelchallenge.github.io for details :)

Want to know what training data has been memorized by models like GPT-4? We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models, without requiring access to 🙅‍♀️ Model weights 🙅‍♀️ Training data 🙅‍♀️ Token probabilities 🧵 (1/5)

A bit of a mess around the conflict of COLM with the ARR (and to lesser degree ICML) reviews release. We feel this is creating a lot of pressure and uncertainty. So, we are pushing our deadlines: Abstracts due March 22 AoE (+48hr) Full papers due March 28 AoE (+24hr) Plz RT 🙏

Introducing 🐻 BEARCUBS 🐻, a “small but mighty” dataset of 111 QA pairs designed to assess computer-using web agents in multimodal interactions on the live web! ✅ Humans achieve 85% accuracy ❌ OpenAI Operator: 24% ❌ Anthropic Computer Use: 14% ❌ Convergence AI Proxy: 13%

Turns out synthetic tasks like needle in a haystack as NOT solved! When we started we thought that English will have perfect performance (given all these green fields heatmaps) but it turns out models struggle with simple task modification! And the gap between langs is HUGE (~34%)!

Huge shoutout to colleagues at Google & Unbabel for extending our WMT24 testset to 55 languages in four domains, this is game changer! 🚀 I really hope it puts the final nail in the coffin of FLORES or WMT14. The field is evolving, legacy testsets can't show your progress arxiv.org/abs/2502.124...

⚠️Current methods for generating instruction-following data fall short for long-range reasoning tasks like narrative claim verification. We present CLIPPER ✂️, a compression-based pipeline that produces grounded instructions for ~$0.5 each, 34x cheaper than human annotations.

How can we generate synthetic data for a task that requires global reasoning over a long context (e.g., verifying claims about a book)? LLMs aren't good at *solving* such tasks, let alone generating data for them. Check out our paper for a compression-based solution!

An absolutely awesome lineup of language pairs for the 20th iteration of WMT 🍾🎉

Guess what? The jubilee 🎉 20th iteration of WMT General MT 🎉 is here, and we want you to participate - as the entry barrier to make an impact is so low! This isn’t just any repeat. We’ve kept what worked, removed what was outdated, and introduced many exciting new twists! Among the key changes are:

New paper that tries to empirically quantify how training-set contamination impacts LLM benchmark scores, by training a variety of 1B and 8B models with controlled types and levels of contamination.

🤔 Interested in how #HCI thinks about using #LLMs, or looking to understand best practices for human-LLM interaction? 🚨🚨New paper: Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review 🧵

No matter how we tried to modify #LLM generated text (paraphrasing, humanization), people who frequently use LLMs for writing are consistently good at detecting model-generated text, though they change cues they rely on! Congrats @jennarussell.bsky.social on first paper!

📢 #nlp #jobs A postdoc for an interdisciplinary project (sociology+NLP), to work on methodology for qualitative interviews with LLMs. This is a NLP position, but some soc/psy background is a plus. Domain PI: @hjalmarcarlsen.bsky.social Apply by March 17 candidate.hr-manager.net/ApplicationI...

📢 The survey 👇 on "intelligence" & ChatGPT is open until Jan 14. All fields are welcome, but a special invitation goes to folks in Philosophy, Psychology, HCI, Cognitive Science, CSS, and AI ethics. Please repost.

We've added #o1 and #Llama 3.3 70B to the #Nocha leaderboard for long-context narrative reasoning! Surprisingly, o1 performs worse than o1-preview, and Llama 3.3 70B matches proprietary models like gpt4o-mini & gemini-Flash. Check out our website for more results! More in 🧵

We're hiring new #nlp faculty this year! Asst or Assoc Professors in NLP at UMass CICS -- careers.umass.edu/amherst/en-u...

OLMo 2 is out 🥳 7B and 13B trained on 5T tokens, and meticulousy instruction tuned using Tulu 3 recipe. Simply the best fully open models yet. Really proud of the work & the amazing team at @ai2.bsky.social

Playing around more with feeds, trying to capture NLP paper threads. Right now it's only looking for ACL Anthology paper links (using a regex included in the feed description), but it could include more. Maybe links + #NLP hashtag? Not sure how to do that though.

📢 NAACL reviews have been released! 🆕 feature alert: ARR now has review issue flagging! Thanks @jkkummerfeld.bsky.social & OR team for help with implementation, and other EiCs for supporting the idea! It'll be live after author response. More details here: aclrollingreview.org/authors#step... /1

Meet Tülu 3, a set of state-of-the-art instruct models with fully open data, eval code, and training algorithms. We invented new methods for fine-tuning language models with RL and built upon best practices to scale synthetic instruction and preference data. Demo, GitHub, paper, and models 👇

Mat is not on 🦋—posting on his behalf! It's time to revisit common assumptions in IR! Embeddings have improved drastically, but mainstream IR evals have stagnated since MSMARCO + BEIR. We ask: on private or tricky IR tasks, are rerankers better? Surely, reranking many docs is best?

Stanford RegLab is hiring predoctoral Research Fellows! The RegLab has been a wonderful place for work spanning from AI ML/NLP/policy to real world impact, successfully working with government organizations. It’s a place to apply technical skills to new areas. www.linkedin.com/posts/reglab...

Steve Johnson on using book text to build a text adventure in NotebookLM. “the game relies on three elements: the original text from my book; a large language model (… Gemini Pro 1.5); and a 400-word prompt that I wrote giving the model instructions on how to host the game” thelongcontext.com

日本で自然言語処理をしている方，ぜひリストにご参加ください． bsky.app/profile/did:...