I knew StackOverflow was in trouble because of LLMs but this graph is insane. It took a decade for Wikipedia to push Encyclopedia Britannica out of print, but only three years for LLMs to make people stop asking questions on Stack Overflow. data.stackexchange.com/stackoverflo...

A student found my personal number and started calling me on WhatsApp to increase their grade on the final😶 Reminding myself of the dumb stuff I did as an 18 year old to find the grace to gently email them that this is inappropriate.

Omg wait. Someone literally posted this paper a couple weeks ago. Good job guys

Following up on my blog post about, I figured I'd create a silly benchmark to test how good LLMs are at playing FizzBuzz for 100 turns. Surprisingly 2 Claude models do well at both the standard game and a slightly modified game where ‘buzz’ should be emitted at multiples of 7 rather than 5… (1/2)

I was thinking about FizzBuzz and LLMs writing code and had a few thoughts. venkatasg.net/blog/fizzbuzz-2...

Presenting a poster with some independent work on dynamic neural audio at 3pm at the AI for Music workshop (room 27)! bostromk.net/ASURA

🥳Life Update! I’m thrilled to share that I’ll be starting as assistant professor for Natural Language Processing @unileipzig.bsky.social in April! I’m deeply grateful to everyone who supported me on this journey. I will be recruiting PhD students with @scadsai.bsky.social, stay tuned for details!

We are accepting submissions for the 25th edition of the Texas Linguistics Society (TLS), a UT Austin grad-student ran Linguistics conference! The conference will run from February 20 - 21, 2026 in Austin. Abstract Deadline: December 17 Notification: January 15

New work to appear @ TACL! Language models (LMs) are remarkably good at generating novel well-formed sentences, leading to claims that they have mastered grammar. Yet they often assign higher probability to ungrammatical strings than to grammatical strings. How can both things be true? 🧵👇

Syntax that spuriously correlates with safe domains can jailbreak LLMs - e.g. below with GPT4o mini Our paper (co w/ Vinith Suriyakumar) on syntax-domain spurious correlations will appear at #NeurIPS2025 as a ✨spotlight! + @marzyehghassemi.bsky.social, @byron.bsky.social, Levent Sagun

"Although I hate leafy vegetables, I prefer daxes to blickets." Can you tell if daxes are leafy vegetables? LM's can't seem to! 📷 We investigate if LMs capture these inferences from connectives when they cannot rely on world knowledge. New paper w/ Daniel, Will, @jessyjli.bsky.social

UT Austin Linguistics is hiring in computational linguistics! Asst or Assoc. We have a thriving group sites.utexas.edu/compling/ and a long proud history in the space. (For instance, fun fact, Jeff Elman was a UT Austin Linguistics Ph.D.) faculty.utexas.edu/career/170793 🤘

Excited to present this at #COLM2025 tomorrow! (Tuesday, 11:00 AM poster session)

I will be giving a short talk on this work at the COLM Interplay workshop on Friday (also to appear at EMNLP)! Will be in Montreal all week and excited to chat about LM interpretability + its interaction with human cognition and ling theory.

I’m excited for COLM this week! Looking forward to chatting with people about interpretability, data efficient training, cog sci and LLM consistency.

The compling group at UT Austin (sites.utexas.edu/compling/) is looking for PhD students! Come join me, @kmahowald.bsky.social, and @jessyjli.bsky.social as we tackle interesting research questions at the intersection of ling, cogsci, and ai! Some topics I am particularly interested in:

I love creating this graph every five years over ACL anthology titles and abstracts. Mentions of nuance/fine-grain seem to be doubling every five years 🙃 Nuance rising has yet to level off among *CL publications.

Accepted at #NeurIPS2025! So proud of Yulu and Dheeraj for leading this! Be on the lookout for more "nuanced yes/no" work from them in the future 👀

Exhibit N on how synthetic text/AI detectors just don't work reliably. Generating some (long) sentences from GPT4.1 and GPT5 with the same prompt, the top open-source model on the RAID benchmark classifies most GPT4.1 outputs as synthetic and most GPT5 as not synthetic.

A brilliant linguist and sociolinguist, RIP Gift article from NYT www.nytimes.com/2025/08/15/u...

The Echoes in AI paper showed quite the opposite with also a story continuation setup. Additionally, we present evidence that both *syntactic* and *discourse* diversity measures show strong homogenization that lexical and cosine used in this paper do not capture.

ACL shortpaper appreciation thread because shortpapers are the best papers. What's the best <=5 pager? I'm nominating an oldie but a goodie

Happy to share the published version of "Art, Understanding, and Mystery"! I often hear some version of the thought that it's bad to understand artworks; this paper attempts to make that claim precise and show one way to defend artistic understanding! journals.publishing.umich.edu/ergo/article...

Decided to finally give programming 'agents'/vibe coding a whirl with something low-stakes/low-risk🤞🏾. Typeproof (built almost entirely with Gemini CLI) lets you view Jonathan Hoefler's (non-pangram) typeface proofs for any Google web font...(1/3) venkatasg.net/typeproof/

🎉 New Benchmark Alert: KRISTEVA – Close‑Reading for LLMs📚 I’m excited to announce a new paper accepted to ACL 2025, in collaboration with Patrick Sui, Philippe Laban, and others!

🇦🇹I'll be at #ACL2025! Recently I've been thinking about: ✨linguistically + cognitively-motivated evals (as always!) ✨understanding multilingualism + representation learning (new!) I'll also be presenting a poster for BehaviorBox on Wed @ Poster Session 4 (Hall 4/5, 10-11:30)!

@simonwillison.net I think you might be the best person to answer this question - but does gemini CLI (free tier) save the output of the chat session somewhere as a log file? Or is it possible to configure that?

Does vision training change how language is represented and used in meaningful ways?🤔The answer is a nuanced yes! Comparing VLM-LM minimal pairs, we find that while the taxonomic organization of the lexicon is similar, VLMs are better at _deploying_ this knowledge. [1/9]

Have there been any re-analysis of the 'Is Google Making Us Stupid' article from 2008 in the news? I'm not saying LLMs are no different from Google, but if we're asking the same question again we need to revisit the old debate and reflect. www.theatlantic.com/magazine/arc...

🦙 how well do LLMs encode discourse knowledge? does that generalize across languages? 🛎️ in our #ACL2025 paper, we uncover fascinating trends about multilingual discourse representations! joint work w/ @florian-eichin.com @barbaraplank.bsky.social @mhedderich.bsky.social 📄 arxiv.org/abs/2503.10515

I'll be at #ICML to present SPRI next week! Come by our poster on Tuesday, July 15, 4:30pm, and let’s catch up on LLM alignment! 😃 🚀TL;DR: We introduce Situated-PRInciples (SPRI), a framework that automatically generates input-specific principles to align responses — with minimal human effort. 🧵

How do language models track mental states of each character in a story, often referred to as Theory of Mind? We reverse-engineered how LLaMA-3-70B-Instruct handles a belief-tracking task and found something surprising: it uses mechanisms strikingly similar to pointer variables in C programming!

Why can’t LMs solve puzzles about the number systems of languages, when they can solve really complex math problems? Our new paper, led by @antararb.bsky.social looks at why this intersection of language and math is difficult, and what this means for LM reasoning! arxiv.org/abs/2506.13886

When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs: 🧵1/9

Language is often strategic, but LLMs tend to play nice. How strategic are they really? Probing into that is key for future safety alignment. 👉Introducing CoBRA🐍, a framework that assesses strategic language. Work with my amazing advisors @jessyjli.bsky.social and @David I. Beaver!

News🗞️ I will return to UT Austin as an Assistant Professor of Linguistics this fall, and join its vibrant community of Computational Linguists, NLPers, and Cognitive Scientists!🤘 Excited to develop ideas about linguistic and conceptual generalization (recruitment details soon!)

A key hypothesis in the history of linguistics is that different constructions share underlying structure. We take advantage of recent advances in mechanistic interpretability to test this hypothesis in Language Models. New work with @kmahowald.bsky.social and @cgpotts.bsky.social! 🧵👇!

🛎️ Excited to announce the 1st Workshop on Bridging NLP and Public Opinion Research at COLM 2025, Oct 10th in Montreal 🇨🇦 As LLMs reshape public discourse and research, collaboration between NLP and Public Opinion Research (POR) is more vital than ever #NLPOR Submit by June 23📄 🔗 tinyurl.com/nlpor25

RIP linguist Haj Ross (1938-2025) en.wikipedia.org/wiki/John_R....

Just published a new paper on how we pragmatically use "I don't know" (and variants) across settings (mediated / face to face), relationship (friend/stranger) and conversation type (task/chat)! #PsychSciSky #linguistics

I'll be presenting this work with @rachelrudinger at #NAACL2025 tomorrow (Wednesday 4/30) in Albuquerque during Session C (Oral/Poster 2) at 2pm! 🔬 Decomposing hypotheses in traditional NLI and defeasible NLI helps us measure various forms of consistency of LLMs. Come join us!

If you're at #NAACL2025 and want to hear about similarity effects for property inheritance in LMs, please stop by! I will be presenting this work on Wednesday at the 11-12:30 poster session on Interpretability & analysis for language models (Hall 3). aclanthology.org/2025.naacl-l...

✨ Attending the ICLR SynthData workshop today (April 27)? synthetic-data-iclr.github.io Lennart Finke (finke.dev) will be presenting our paper "Parameterized Synthetic Text Generation with SimpleStories" at the 11:30-12:30PM poster session! openreview.net/forum?id=JO8... More details below!👇

Evaluating language model responses on open-ended tasks is hard! 🤔 We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️. EvalAgent identifies 👩‍🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇

Have that eerie feeling of déjà vu when reading model-generated text 👀, but can’t pinpoint the specific words or phrases 👀? ✨We introduce QUDsim, to quantify discourse similarities beyond lexical, syntactic, and content overlap.

Our final South by Semantics lecture at UT Austin is happening on Wednesday April 23!

I might be able to hire a postdoc for this fall in computational linguistics at UT Austin. Topics in the general LLM + cognitive space (particularly reasoning, chain of thought, LLMs + code) and LLM + linguistic space. If this could be of interest, feel free to get in touch!

One of the ways that LLMs can be inconsistent is the "generator-validator gap," where LLMs deem their own answers incorrect. 🎯 We demonstrate that ranking-based discriminator training can significantly reduce this gap, and improvements on one task often generalize to others! 🧵👇

Congrats to @vshirazy.bsky.social for winning a NSF GRFP! Venus is writing a fascinating honors thesis on Farsi complex predicates, and has long been my editorial assistant at Language. She's amazing, and I know she'll excel at her next adventure in the Linguistics PhD program at U. of Chicago!