Nice research! You may be interested in the small scale ($4 budget) verification performed by my personal Opus agent here: muninn.austegard.com/blog/this-tr... in which we also introduced a framing-resistant prompt to see how much that would mitigate the effetcs. 1/3

If you ask the same question with different framing/phrasing, do language models change their answers? This is super important in medicine because different info can have real consequences! Check out this new work from @hyesunyun.bsky.social

Heading to #EACL2026! 🇲🇦 Friday 11a Poster Session 6: LMs struggle to perform inferences around discourse connectives aclanthology.org/2026.eacl-lo... Sunday 5p TeachingNLP workshop: new course on discourse+generation aclanthology.org/2026.teachin... w/ @kanishka.bsky.social + Daniel Brubaker

Want to know how well the models can brainstorm connections across different concepts? Super excited about @manyawadhwa.bsky.social’s work on measuring associative creativity!

What is the interplay between representations learned from (language) surface forms alone, and those learned from more grounded evidence (e.g.,vision)? Excited to share new work understanding “Cross-modal taxonomic generalization” in (V)LMs arxiv.org/abs/2603.07474 1/

Check out our special theme: new missions for NLP research!

Nearly 2 years ago, @jessyjli.bsky.social, @janetlauyeung.bsky.social, @valentinapy.bsky.social, and I decided that it's time to bring discourse structure to the center of NLP teaching.

Check out @asher-zheng.bsky.social's work on quantifying strategic language in dialogue, just appeared in the Dialogue and Discourse journal. We study non-cooperative moves that are subtle to capture, where modern AI still have trouble comprehending. Work w/ David_Beaver

“All bears have a property”, “Some bears have a property”, “Bears have a property” are different in terms of how the property is generalized to a specific bear – a great example of how language constrains thought! This holds for kids, adults, and according to our new work, (V)LMs! 🧵

🚨Be careful with LLMs when you ask health related questions -- even when the model relies on "evidence"! Kaijie's paper reveals a key weakness and the tricky balance between safety and faithfulness 👉

Accepted at EACL - excited about Morocco!

New work to appear @ TACL! Language models (LMs) are remarkably good at generating novel well-formed sentences, leading to claims that they have mastered grammar. Yet they often assign higher probability to ungrammatical strings than to grammatical strings. How can both things be true? 🧵👇

Incredibly honored to serve as #EMNLP 2026 Program Chair along with @sunipadev.bsky.social and Hung-yi Lee, and General Chair @andre-t-martins.bsky.social. Looking forward to Budapest!! (With thanks to Lisa Chuyuan Li who took this photo in Suzhou!)

Delighted Sasha's (first year PhD!) work using mech interp to study complex syntax constructions won an Outstanding Paper Award at EMNLP! Also delighted the ACL community continues to recognize unabashedly linguistic topics like filler-gaps... and the huge potential for LMs to inform such topics!

Think your LLMs “understand” words like although/but/therefore? Think again! They perform at chance for making inferences from certain discourse connectives expressing concession

🚨 Does your LLM really understand code -- or is it just really good at remembering it? We built **PLSemanticsBench** to find out. The results: a wild mix. ✅The Brilliant: Top reasoning models can execute complex, fuzzer-generated programs -- even with 5+ levels of nested loops! 🤯 ❌The Brittle: 🧵

Find my students and collaborators at COLM this week! Tuesday morning: @juand-r.bsky.social and @ramyanamuduri.bsky.social 's papers (find them if you missed it!) Wednesday pm: @manyawadhwa.bsky.social 's EvalAgent Thursday am: @anirudhkhatry.bsky.social 's CRUST-Bench oral spotlight + poster

We’re hiring faculty as well! Happy to talk about it at COLM!

Can we quantify what makes some text read like AI "slop"? We tried 👇

I’m at #COLM2025 from Wed with: @siyuansong.bsky.social Tue am introspection arxiv.org/abs/2503.07513 @qyao.bsky.social Wed am controlled rearing: arxiv.org/abs/2503.20850 @sashaboguraev.bsky.social INTERPLAY ling interp: arxiv.org/abs/2505.16002 I’ll talk at INTERPLAY too. Come say hi!

On my way to #COLM2025 🍁 Check out jessyli.com/colm2025 QUDsim: Discourse templates in LLM stories arxiv.org/abs/2504.09373 EvalAgent: retrieval-based eval targeting implicit criteria arxiv.org/abs/2504.15219 RoboInstruct: code generation for robotics with simulators arxiv.org/abs/2405.20179

Traveling to my first @colmweb.org🍁 Not presenting anything but here are two posters you should visit: 1. @qyao.bsky.social on Controlled rearing for direct and indirect evidence for datives (w/ me, @weissweiler.bsky.social and @kmahowald.bsky.social), W morning Paper: arxiv.org/abs/2503.20850

All of us (@kanishka.bsky.social @kmahowald.bsky.social and me) are looking for PhD students this cycle! If computational linguistics/NLP is your passion, join us at UT Austin! For my areas see jessyli.com

Can AI aid scientists amidst their own workflows, when they do not know step-by-step workflows and may not know, in advance, the kinds of scientific utility a visualization would bring? Check out @sebajoe.bsky.social’s feature on ✨AstroVisBench:

📣 NEW HCTS course developed in collaboration with @tephi-tx.bsky.social: AI in Health Communication 📣 Explore responsible applications and best practices for maximizing impact and building trust with @utaustin.bsky.social experts @jessyjli.bsky.social & @mackert.bsky.social. 💻: rebrand.ly/HCTS_AI

long range narrative understanding, even basic fact checking that humans easily get near perfect on, has barely improved in LMs over years novelchallenge.github.io

🤖 🧠 NEW PAPER ON COGSCI & AI 🧠 🤖 Recent neural networks capture properties long thought to require symbols: compositionality, productivity, rapid learning So what role should symbols play in theories of the mind? For our answer...read on! Paper: arxiv.org/abs/2508.05776 1/n

I agree this thread's headline claim seems premature. Let me add our recent ACL Findings paper, with Dexter Ju and @hagenblix.bsky.social, which found syntactic simplification in at least some LMs, in a novel domain regeneration setting: aclanthology.org/2025.finding...

The Echoes in AI paper showed quite the opposite with also a story continuation setup. Additionally, we present evidence that both *syntactic* and *discourse* diversity measures show strong homogenization that lexical and cosine used in this paper do not capture.

Tuesday at #ACL2025: Jan will be presenting this from 4-5:30pm in x4/x5! Turns out content selection in LLMs are highly consistent with each other, but not so much with their own notion of importance or with human’s…

Looking forward to attending #cogsci2025 (Jul 29 - Aug 3)! I’m especially excited to meet students who will be applying to PhD programs in Computational Ling/CogSci in the coming cycle. Please reach out if you want to meet up and chat! Email is the best way, but DM also works if you must! quick🧵:

If you’re heading to ICML, check out Hongli’s work on context-specific alignment!

Check out this new opinion piece from Sebastian and Lily! We have really powerful AI systems now, so what’s the bottleneck preventing the wider adoption of fact checking systems, in high stakes scenarios like medicine? It’s how we define the tasks 👇

We have very good frameworks for cooperative dialog… but how about the opposite? @asher-zheng.bsky.social’s new paper takes a game-theoretic view and develops new metrics to quantify non-cooperative language ♟️ Turns out LLMs don’t have the pragmatic capabilities to perceive these…

Congrats to Sebastian et al. This is our first fully within-CosmicAI research project, setting the foundation for our future work.

Is AI ready to play a real role in science? This work with @nsfsimonscosmicai.bsky.social evaluates LLMs targeting the implementation of scientific workflows, and the scientific utility of visualizations from LLM-generated code -- and the answer is not yet, even with the best SOTA models 👇

Super thrilled that @kanishka.bsky.social is going to join UT Linguistics as our newest computational linguistics faculty member -- looking forward to doing great research together! 🧑‍🎓Students: Kanishka is a GREAT mentor -- apply to be his PhD student in the upcoming cycle!!

To appear #ICML2025!! 🎉

Where are those implicit, nuanced, and specific criteria in the land of LLM evaluation? Manya’s EvalAgent gets them!

We finally have a way to quantify that "LLM vibe" you feel about generated texts... There *are* tangible discourse similarities across models! Check out QUDsim👇

Have that eerie feeling of déjà vu when reading model-generated text 👀, but can’t pinpoint the specific words or phrases 👀? ✨We introduce QUDsim, to quantify discourse similarities beyond lexical, syntactic, and content overlap.

Team member highlight! Jessy Li @jessyjli.bsky.social associate professor of computational linguistics and NLP at @utaustin.bsky.social's Linguistics department, will be the AI lead of the Explorable Universe team. The team's goal is to develop generative AI models for astronomy research. #cosmicai

🌟Job ad🌟 We (@gregdnlp.bsky.social, @mattlease.bsky.social and I) are hiring a postdoc fellow within the CosmicAI Institute, to do galactic work with LLMs and generative AI! If you would like to push the frontiers of foundation models to help solve myths of the universe, please apply!

Do you want to know what information LLMs prioritize in text synthesis tasks? Here's a short 🧵 about our new paper, led by Jan Trienes: an interpretable framework for salience analysis in LLMs. First of all, information salience is a fuzzy concept. So how can we even measure it? (1/6)

🚨 Do LLMs fall for spin in medical literature? 🤔 In our new preprint, we find that LLMs are susceptible to biased reporting of clinical treatment benefits in abstracts—more so than human experts. 📄🔍 [1/7] Full Paper: arxiv.org/abs/2502.07963 🧵👇

The principles that LLMs align with should be specific to the task at hand! Check out @hongli-zhan.bsky.social’s latest work 👇

1. Hey NAACL #NLP #NLProc members please VOTE for the 25-26 exec! We have an amazing lineup of candidates, see statements here naacl.org/elections/20... How to vote: current members look for an email from Election Runner!

A starter pack for the NLP and Computational Linguistics researchers at UT Austin! go.bsky.app/75g9JLT

We at UT Linguistics are hiring for 🔥 2 faculty positions in Computational Linguistics! Assistant or Associate professors, deadline Dec 1. UT has a super vibrant comp ling & #nlp community!! Apply here 👉 apply.interfolio.com/158280

All the ACL chapters are here now: @aaclmeeting.bsky.social @emnlpmeeting.bsky.social @eaclmeeting.bsky.social @naaclmeeting.bsky.social #NLProc