🚨New paper on AI & copyright Authors have sued LLM companies for using books w/o permission for model training. Courts however need empirical evidence of market harm. Our preregistered study exactly addresses this gap. Joint work w Jane Ginsburg from Columbia Law and @dhillonp.bsky.social 1/n🧵

Well this is sure to be a blockbuster AI article... @jennarussell.bsky.social et al are kicking ass and taking names in journalism, both individuals and organizations. "AI use in American newspapers is widespread, uneven, and rarely disclosed" arxiv.org/abs/2510.18774

AI is infiltrating American newsrooms. Sadly, it is mostly *undisclosed* meaning that readers are often unaware that they are consuming LLM text. Even worse, we find some of these texts making it to the print press (undisclosed) Can we at least be honest about using models for editing?

AI is already at work in American newsrooms. We examine 186k articles published this summer and find that ~9% are either fully or partially AI-generated, usually without readers having any idea. Here's what we learned about how AI is influencing local and national journalism:

📢 Announcing the First Workshop on Multilingual and Multicultural Evaluation (MME) at #EACL2026 🇲🇦 MME focuses on resources, metrics & methodologies for evaluating multilingual systems! multilingual-multicultural-evaluation.github.io 📅 Workshop Mar 24–29, 2026 🗓️ Submit by Dec 19, 2025

I'd love to see someone try to estimate just how much time and money has gone into research that is either fully undermined by reliance on LLMs or fully pointless --- because obvious if you start from an understanding of what LLMs actually are. www.pnas.org/doi/10.1073/...

I'm not sure why people lost the ability to do related work properly but if you absolutely need to use AI at least proofread it? (And they most likely edited with ai) www.pangram.com/history/01bf...

The viral "Definition of AGI" paper tells you to read fake references which do not exist! Proof: different articles present at the specified journal/volume/page number, and their titles exist nowhere on any searchable repository. Take this as a warning to not use LMs to generate your references!

𝑵𝒆𝒘 𝒃𝒍𝒐𝒈𝒑𝒐𝒔𝒕! A rundown of some cool papers I got to chat about at #COLM2025 and some scattered thoughts saxon.me/blog/2025/co...

Come to talk with us today about the evaluation of long form multilingual generation at the second poster session #COLM2025 📍4:30–6:30 PM / Room 710 – Poster #8

Off to #COLM fake Fuji looks really good today. 本物は下からしか見たことがないが、今日は少なくとも偽物が上から見えて嬉しい。

When reading AI reasoning text (aka CoT), we (humans) form a narrative about the underlying computation process, which we take as a transparent explanation of model behavior. But what if our narratives are wrong? We measure that and find it usually is. Now on arXiv: arxiv.org/abs/2508.16599

📊 Preliminary ranking of WMT 2025 General Machine Translation benchmark is here! But don't draw conclusions just yet - automatic metrics are biased for techniques like metric as a reward model or MBR. The official human ranking will be part of General MT findings at WMT. arxiv.org/abs/2508.14909

Happy to see this work accepted to #EMNLP2025! 🎉🎉🎉

✨We are thrilled to announce that over 3200 papers have been accepted to #EMNLP2025 ✨ This includes over 1800 main conference papers and over 1400 papers in findings! Congratulations to all authors!! 🎉🎉🎉

The Echoes in AI paper showed quite the opposite with also a story continuation setup. Additionally, we present evidence that both *syntactic* and *discourse* diversity measures show strong homogenization that lexical and cosine used in this paper do not capture.

GPT-5 lands first place on NoCha, our long-context book understanding benchmark. That said, this is a tiny improvement (~1%) over o1-preview, which was released almost one year ago. Have long-context models hit a wall? Accuracy of human readers is >97%... Long way to go!

🗓️29 July, 4 PM: Automated main concept generation for narrative discourse assessment in aphasia. w/ @marisahudspeth.bsky.social, Polly Stokes, Jacquie Kurland, and @brenocon.bsky.social 📍Hall 4/5. Come by to chat about argumentation, narrative texts, policy & law, and beyond! #ACL2025NLP

Excited to present two papers at #ACL2025! 🗓️30 July, 11 AM: 𝛿-Stance: A Large-Scale Real World Dataset of Stances in Legal Argumentation. w/ Douglas Rice and @brenocon.bsky.social 📍At Hall 4/5. 🧵👇

📣 Life update: Thrilled to announce that I’ll be starting as faculty at the Max Planck Institute for Software Systems this Fall! I’ll be recruiting PhD students in the upcoming cycle, as well as research interns throughout the year: lasharavichander.github.io/contact.html

For EMNLP 2025’s special theme of "Advancing our Reach: Interdisciplinary Recontextualization of NLP", we are organizing a panel of experts, and would like input from the community at large as we prepare. Please take a moment to fill in this survey: forms.office.com/r/pWFFA0Gss1

A new definition for AGI just dropped, and it is a bad one.

Now accepted to #COLM2025 @colmweb.org 🇨🇦🎉

What should Machine Translation research look like in the age of multilingual LLMs? Here’s one answer from researchers across NLP/MT, Translation Studies, and HCI. "An Interdisciplinary Approach to Human-Centered Machine Translation" arxiv.org/abs/2506.13468

Extremely interesting new task that gives a model a literary text, plus a critical essay about it — with one quotation masked. Can the model figure out which quotation from the original work would support these claims? Best-performing models exceed human readers. #MLSky arxiv.org/abs/2506.030...

Tired of AI slop? Our work on "Frankentexts" shows how LLMs can stitch together random fragments of human writing into coherent, relevant responses to arbitrary prompts. Frankentexts are weirdly creative, and they also pose problems for AI detectors: are they AI? human? More 👇

🤔 What if you gave an LLM thousands of random human-written paragraphs and told it to write something new -- while copying 90% of its output from those texts? 🧟 You get what we call a Frankentext! 💡 Frankentexts are surprisingly coherent and tough for AI detectors to flag.

Interested in crosslingual memorization? Check out our new work :) Congrats to Emir, Alisha, and Minh for putting together their first research paper 🎉

LLMs memorize novels 📚 in English. But what about existing translations? Or translations into new languages? Our 🦉OWL dataset (31K/10 languages) shows GPT4o recognizes books: 92% English 83% official translations 69% unseen translations 75% as audio (EN)

That's just insane! @naaclmeeting.bsky.social Link to original report: www.lemonde.fr/en/internati...

We have also added #Maverick which is arguably better but (similarly to other models) struggles when the evidence is not easy to retrieve (ie requires more reasoning than simple retrieval -- hard set). More here: novelchallenge.github.io

Llama 4's massive context window is impressive! However, the best Llama model for long-context understanding over books is still Llama 3.1 405B. Llama 4 Scout is especially bad at our NoCha benchmark, performing below random chance.

We have updated #nocha (long-context benchmark measuring how well models process book-length narratives) with #Llama4 Scout. Sadly, the performance was below the random level, much lower than the reported model's performance on a retrieval task (needle in the haystack). novelchallenge.github.io

We have updated #nocha, a leaderboard for reasoning over long-context narratives 📖, with some new models including #Gemini 2.5 Pro which shows massive improvements over the previous version! Congrats to #Gemini team 🪄 🧙 Check 🔗 novelchallenge.github.io for details :)

Want to know what training data has been memorized by models like GPT-4? We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models, without requiring access to 🙅‍♀️ Model weights 🙅‍♀️ Training data 🙅‍♀️ Token probabilities 🧵 (1/5)

A bit of a mess around the conflict of COLM with the ARR (and to lesser degree ICML) reviews release. We feel this is creating a lot of pressure and uncertainty. So, we are pushing our deadlines: Abstracts due March 22 AoE (+48hr) Full papers due March 28 AoE (+24hr) Plz RT 🙏

Introducing 🐻 BEARCUBS 🐻, a “small but mighty” dataset of 111 QA pairs designed to assess computer-using web agents in multimodal interactions on the live web! ✅ Humans achieve 85% accuracy ❌ OpenAI Operator: 24% ❌ Anthropic Computer Use: 14% ❌ Convergence AI Proxy: 13%

Turns out synthetic tasks like needle in a haystack as NOT solved! When we started we thought that English will have perfect performance (given all these green fields heatmaps) but it turns out models struggle with simple task modification! And the gap between langs is HUGE (~34%)!

Huge shoutout to colleagues at Google & Unbabel for extending our WMT24 testset to 55 languages in four domains, this is game changer! 🚀 I really hope it puts the final nail in the coffin of FLORES or WMT14. The field is evolving, legacy testsets can't show your progress arxiv.org/abs/2502.124...

⚠️Current methods for generating instruction-following data fall short for long-range reasoning tasks like narrative claim verification. We present CLIPPER ✂️, a compression-based pipeline that produces grounded instructions for ~$0.5 each, 34x cheaper than human annotations.

How can we generate synthetic data for a task that requires global reasoning over a long context (e.g., verifying claims about a book)? LLMs aren't good at *solving* such tasks, let alone generating data for them. Check out our paper for a compression-based solution!

An absolutely awesome lineup of language pairs for the 20th iteration of WMT 🍾🎉

Guess what? The jubilee 🎉 20th iteration of WMT General MT 🎉 is here, and we want you to participate - as the entry barrier to make an impact is so low! This isn’t just any repeat. We’ve kept what worked, removed what was outdated, and introduced many exciting new twists! Among the key changes are:

New paper that tries to empirically quantify how training-set contamination impacts LLM benchmark scores, by training a variety of 1B and 8B models with controlled types and levels of contamination.

🤔 Interested in how #HCI thinks about using #LLMs, or looking to understand best practices for human-LLM interaction? 🚨🚨New paper: Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review 🧵

No matter how we tried to modify #LLM generated text (paraphrasing, humanization), people who frequently use LLMs for writing are consistently good at detecting model-generated text, though they change cues they rely on! Congrats @jennarussell.bsky.social on first paper!

📢 #nlp #jobs A postdoc for an interdisciplinary project (sociology+NLP), to work on methodology for qualitative interviews with LLMs. This is a NLP position, but some soc/psy background is a plus. Domain PI: @hjalmarcarlsen.bsky.social Apply by March 17 candidate.hr-manager.net/ApplicationI...

📢 The survey 👇 on "intelligence" & ChatGPT is open until Jan 14. All fields are welcome, but a special invitation goes to folks in Philosophy, Psychology, HCI, Cognitive Science, CSS, and AI ethics. Please repost.

We've added #o1 and #Llama 3.3 70B to the #Nocha leaderboard for long-context narrative reasoning! Surprisingly, o1 performs worse than o1-preview, and Llama 3.3 70B matches proprietary models like gpt4o-mini & gemini-Flash. Check out our website for more results! More in 🧵