Syntax that spuriously correlates with safe domains can jailbreak LLMs - e.g. below with GPT4o mini Our paper (co w/ Vinith Suriyakumar) on syntax-domain spurious correlations will appear at #NeurIPS2025 as a ✨spotlight! + @marzyehghassemi.bsky.social, @byron.bsky.social, Levent Sagun

"Although I hate leafy vegetables, I prefer daxes to blickets." Can you tell if daxes are leafy vegetables? LM's can't seem to! 📷 We investigate if LMs capture these inferences from connectives when they cannot rely on world knowledge. New paper w/ Daniel, Will, @jessyjli.bsky.social

UT Austin Linguistics is hiring in computational linguistics! Asst or Assoc. We have a thriving group sites.utexas.edu/compling/ and a long proud history in the space. (For instance, fun fact, Jeff Elman was a UT Austin Linguistics Ph.D.) faculty.utexas.edu/career/170793 🤘

Excited to present this at #COLM2025 tomorrow! (Tuesday, 11:00 AM poster session)

I will be giving a short talk on this work at the COLM Interplay workshop on Friday (also to appear at EMNLP)! Will be in Montreal all week and excited to chat about LM interpretability + its interaction with human cognition and ling theory.

I’m excited for COLM this week! Looking forward to chatting with people about interpretability, data efficient training, cog sci and LLM consistency.

The compling group at UT Austin (sites.utexas.edu/compling/) is looking for PhD students! Come join me, @kmahowald.bsky.social, and @jessyjli.bsky.social as we tackle interesting research questions at the intersection of ling, cogsci, and ai! Some topics I am particularly interested in:

I love creating this graph every five years over ACL anthology titles and abstracts. Mentions of nuance/fine-grain seem to be doubling every five years 🙃 Nuance rising has yet to level off among *CL publications.

Accepted at #NeurIPS2025! So proud of Yulu and Dheeraj for leading this! Be on the lookout for more "nuanced yes/no" work from them in the future 👀

Exhibit N on how synthetic text/AI detectors just don't work reliably. Generating some (long) sentences from GPT4.1 and GPT5 with the same prompt, the top open-source model on the RAID benchmark classifies most GPT4.1 outputs as synthetic and most GPT5 as not synthetic.

A brilliant linguist and sociolinguist, RIP Gift article from NYT www.nytimes.com/2025/08/15/u...

The Echoes in AI paper showed quite the opposite with also a story continuation setup. Additionally, we present evidence that both *syntactic* and *discourse* diversity measures show strong homogenization that lexical and cosine used in this paper do not capture.

ACL shortpaper appreciation thread because shortpapers are the best papers. What's the best <=5 pager? I'm nominating an oldie but a goodie

Happy to share the published version of "Art, Understanding, and Mystery"! I often hear some version of the thought that it's bad to understand artworks; this paper attempts to make that claim precise and show one way to defend artistic understanding! journals.publishing.umich.edu/ergo/article...

Decided to finally give programming 'agents'/vibe coding a whirl with something low-stakes/low-risk🤞🏾. Typeproof (built almost entirely with Gemini CLI) lets you view Jonathan Hoefler's (non-pangram) typeface proofs for any Google web font...(1/3) venkatasg.net/typeproof/

🎉 New Benchmark Alert: KRISTEVA – Close‑Reading for LLMs📚 I’m excited to announce a new paper accepted to ACL 2025, in collaboration with Patrick Sui, Philippe Laban, and others!

🇦🇹I'll be at #ACL2025! Recently I've been thinking about: ✨linguistically + cognitively-motivated evals (as always!) ✨understanding multilingualism + representation learning (new!) I'll also be presenting a poster for BehaviorBox on Wed @ Poster Session 4 (Hall 4/5, 10-11:30)!

@simonwillison.net I think you might be the best person to answer this question - but does gemini CLI (free tier) save the output of the chat session somewhere as a log file? Or is it possible to configure that?

Does vision training change how language is represented and used in meaningful ways?🤔The answer is a nuanced yes! Comparing VLM-LM minimal pairs, we find that while the taxonomic organization of the lexicon is similar, VLMs are better at _deploying_ this knowledge. [1/9]

Have there been any re-analysis of the 'Is Google Making Us Stupid' article from 2008 in the news? I'm not saying LLMs are no different from Google, but if we're asking the same question again we need to revisit the old debate and reflect. www.theatlantic.com/magazine/arc...

🦙 how well do LLMs encode discourse knowledge? does that generalize across languages? 🛎️ in our #ACL2025 paper, we uncover fascinating trends about multilingual discourse representations! joint work w/ @florian-eichin.com @barbaraplank.bsky.social @mhedderich.bsky.social 📄 arxiv.org/abs/2503.10515

I'll be at #ICML to present SPRI next week! Come by our poster on Tuesday, July 15, 4:30pm, and let’s catch up on LLM alignment! 😃 🚀TL;DR: We introduce Situated-PRInciples (SPRI), a framework that automatically generates input-specific principles to align responses — with minimal human effort. 🧵

How do language models track mental states of each character in a story, often referred to as Theory of Mind? We reverse-engineered how LLaMA-3-70B-Instruct handles a belief-tracking task and found something surprising: it uses mechanisms strikingly similar to pointer variables in C programming!

Why can’t LMs solve puzzles about the number systems of languages, when they can solve really complex math problems? Our new paper, led by @antararb.bsky.social looks at why this intersection of language and math is difficult, and what this means for LM reasoning! arxiv.org/abs/2506.13886

When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs: 🧵1/9

Language is often strategic, but LLMs tend to play nice. How strategic are they really? Probing into that is key for future safety alignment. 👉Introducing CoBRA🐍, a framework that assesses strategic language. Work with my amazing advisors @jessyjli.bsky.social and @David I. Beaver!

News🗞️ I will return to UT Austin as an Assistant Professor of Linguistics this fall, and join its vibrant community of Computational Linguists, NLPers, and Cognitive Scientists!🤘 Excited to develop ideas about linguistic and conceptual generalization (recruitment details soon!)

A key hypothesis in the history of linguistics is that different constructions share underlying structure. We take advantage of recent advances in mechanistic interpretability to test this hypothesis in Language Models. New work with @kmahowald.bsky.social and @cgpotts.bsky.social! 🧵👇!

🛎️ Excited to announce the 1st Workshop on Bridging NLP and Public Opinion Research at COLM 2025, Oct 10th in Montreal 🇨🇦 As LLMs reshape public discourse and research, collaboration between NLP and Public Opinion Research (POR) is more vital than ever #NLPOR Submit by June 23📄 🔗 tinyurl.com/nlpor25

RIP linguist Haj Ross (1938-2025) en.wikipedia.org/wiki/John_R....

Just published a new paper on how we pragmatically use "I don't know" (and variants) across settings (mediated / face to face), relationship (friend/stranger) and conversation type (task/chat)! #PsychSciSky #linguistics

I'll be presenting this work with @rachelrudinger at #NAACL2025 tomorrow (Wednesday 4/30) in Albuquerque during Session C (Oral/Poster 2) at 2pm! 🔬 Decomposing hypotheses in traditional NLI and defeasible NLI helps us measure various forms of consistency of LLMs. Come join us!

If you're at #NAACL2025 and want to hear about similarity effects for property inheritance in LMs, please stop by! I will be presenting this work on Wednesday at the 11-12:30 poster session on Interpretability & analysis for language models (Hall 3). aclanthology.org/2025.naacl-l...

✨ Attending the ICLR SynthData workshop today (April 27)? synthetic-data-iclr.github.io Lennart Finke (finke.dev) will be presenting our paper "Parameterized Synthetic Text Generation with SimpleStories" at the 11:30-12:30PM poster session! openreview.net/forum?id=JO8... More details below!👇

Evaluating language model responses on open-ended tasks is hard! 🤔 We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️. EvalAgent identifies 👩‍🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇

Have that eerie feeling of déjà vu when reading model-generated text 👀, but can’t pinpoint the specific words or phrases 👀? ✨We introduce QUDsim, to quantify discourse similarities beyond lexical, syntactic, and content overlap.

Our final South by Semantics lecture at UT Austin is happening on Wednesday April 23!

I might be able to hire a postdoc for this fall in computational linguistics at UT Austin. Topics in the general LLM + cognitive space (particularly reasoning, chain of thought, LLMs + code) and LLM + linguistic space. If this could be of interest, feel free to get in touch!

One of the ways that LLMs can be inconsistent is the "generator-validator gap," where LLMs deem their own answers incorrect. 🎯 We demonstrate that ranking-based discriminator training can significantly reduce this gap, and improvements on one task often generalize to others! 🧵👇

Congrats to @vshirazy.bsky.social for winning a NSF GRFP! Venus is writing a fascinating honors thesis on Farsi complex predicates, and has long been my editorial assistant at Language. She's amazing, and I know she'll excel at her next adventure in the Linguistics PhD program at U. of Chicago!

[📄] Are LLMs mindless token-shifters, or do they build meaningful representations of language? We study how LLMs copy text in-context, and physically separate out two types of induction heads: token heads, which copy literal tokens, and concept heads, which copy word meanings.

With ChatGPT's o1 model (using "advanced reasoning").

LMs learn argument-based preferences for dative constructions (preferring recipient first when it’s shorter), consistent with humans. Is this from memorizing preferences in training? New paper w/ @kanishka.bsky.social , @weissweiler.bsky.social , @kmahowald.bsky.social arxiv.org/abs/2503.20850

Introducing ParaSpeechCaps, our large-scale style captions dataset that enables rich, expressive control for text-to-speech models! Beyond basic pitch or speed controls, our models can generate speech that sounds "guttural", "scared", "whispered" and more; 59 style tags in total. 🧵👇

🌟Job ad🌟 We (@gregdnlp.bsky.social, @mattlease.bsky.social and I) are hiring a postdoc fellow within the CosmicAI Institute, to do galactic work with LLMs and generative AI! If you would like to push the frontiers of foundation models to help solve myths of the universe, please apply!

✨New paper✨ Linguistic evaluations of LLMs often implicitly assume that language is generated by symbolic rules. In a new position paper, @adelegoldberg.bsky.social, @kmahowald.bsky.social and I argue that languages are not Lego sets, and evaluations should reflect this! arxiv.org/pdf/2502.13195

Constitutional AI works great for aligning LLMs, but the principles can be too generic to apply. Can we guide responses with context-situated principles instead? Introducing SPRI, a system that produces principles tailored to each query, with minimal to no human effort. arxiv.org/pdf/2502.03397

not Lynch's movie, I know, but god, what a sendoff

Thanks to all for your kind remembrances on this awful day. Words will come later. Only feelings at the moment. Mourn and remember him but don’t forget to celebrate too. We won’t see his like again. The man from another place has gone home. #DavidLynch #TwinPeaks