𝙒𝙚'𝙧𝙚 𝙝𝙞𝙧𝙞𝙣𝙜 𝙣𝙚𝙬 𝙛𝙖𝙘𝙪𝙡𝙩𝙮 𝙢𝙚𝙢𝙗𝙚𝙧𝙨! KSoC: utah.peopleadmin.com/postings/190... (AI broadly) Education + AI: - utah.peopleadmin.com/postings/189... - utah.peopleadmin.com/postings/190... Computer Vision: - utah.peopleadmin.com/postings/183...

🎉 Congratulations to all #EMNLP2025 award winners 🎉 Starting with the ✨Best Paper award ✨: "Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index" by Hao Xu, Jiacheng Liu, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi aclanthology.org/2025.emnlp-m... 1/n

Thrilled to see this work recognized at #EMNLP2025! This framework and approach to measuring CoT faithfulness have been hugely influential for how I think about reasoning evaluation, and I'm so lucky to have worked with such brilliant collaborators. Huge credit to @mtutek.bsky.social

Very honored to be one out of seven outstanding papers at this years' EMNLP :) Huge thanks to my amazing collaborators @fatemehc.bsky.social @anamarasovic.bsky.social @boknilev.bsky.social , this would not have been possible without them!

Check out Martin's talk at #EMNLP2025 today (Wed)! If you care about CoT faithfulness, you 𝘮𝘶𝘴𝘵 read this paper. It introduces the first method for measuring CoT faithfulness that is not purely behavioral, but operates with the internals!

Go check Alex's poster today (Wed) in Suzhou! #EMNLP2025 I'm still so proud of our work (led by @lasha.bsky.social) on CondaQA, so we had to ask what would happen if we tried to create high-quality reasoning-over-text benchmarks now that LLMs are available. Turns out, we'd make an easier benchmark!

Flying out to @emnlpmeeting soon🇨🇳 I'll present our parametric CoT faithfulness work (arxiv.org/abs/2502.14829) on Wednesday at the second Interpretability session, 16:30-18:00 local time A104-105 If you're in Suzhou, reach out to talk all things reasoning :)

Sort of starting to believe that we really do need academic metrics that punish publishing too much

I'll be in Suzhou 🇨🇳 at #EMNLP this week presenting "What has been Lost with Synthetic Evaluation?" done with @anamarasovic.bsky.social & @lasha.bsky.social! 🎉 📍Findings Session 1 - Hall C 📅 Wed, November 5, 13:00 - 14:00 arxiv.org/abs/2505.22830

🧠 Can large language models build the very benchmarks used to evaluate them? In “What Has Been Lost with Synthetic Evaluation”, Ana Marasović (@anamarasovic.bsky.social) and collaborators ask what happens when LLMs start generating the datasets used to test their reasoning. (1/6🧵)

👉 Do large language models really reason the way their chain-of-thoughts suggest? This week on #WiAIRpodcast, we talk with Ana Marasović (@anamarasovic.bsky.social) about her paper “Chain-of-Thought Unfaithfulness as Disguised Accuracy.” (1/6🧵) 📄 Paper: arxiv.org/pdf/2402.14897

Can you trust your reward model alignment scores? New work presented today at the COLM Workshop on Socially Responsible Language Modelling Research led by Purbid Bambroo and in collaboration with @anamarasovic.bsky.social that probes LLM preference test sets for redundancy and inflated scores. 1/8

📣Tomorrow at #COLM2025: 1️⃣ Purbid's 𝐩𝐨𝐬𝐭𝐞𝐫 at 𝐒𝐨𝐋𝐚𝐑 (𝟏𝟏:𝟏𝟓𝐚𝐦-𝟏:𝟎𝟎𝐩𝐦) on catching redundant preference pairs & how pruning them hurts accuracy; www.anamarasovic.com/publications... 2️⃣ My 𝐭𝐚𝐥𝐤 at 𝐗𝐋𝐋𝐌-𝐑𝐞𝐚𝐬𝐨𝐧-𝐏𝐥𝐚𝐧 (𝟏𝟐𝐩𝐦) on measuring CoT faithfulness by looking at internals, not just behaviorally 1/3

📣Tomorrow at #COLM2025: 1️⃣ Purbid's 𝐩𝐨𝐬𝐭𝐞𝐫 at 𝐒𝐨𝐋𝐚𝐑 (𝟏𝟏:𝟏𝟓𝐚𝐦-𝟏:𝟎𝟎𝐩𝐦) on catching redundant preference pairs & how pruning them hurts accuracy; www.anamarasovic.com/publications... 2️⃣ My 𝐭𝐚𝐥𝐤 at 𝐗𝐋𝐋𝐌-𝐑𝐞𝐚𝐬𝐨𝐧-𝐏𝐥𝐚𝐧 (𝟏𝟐𝐩𝐦) on measuring CoT faithfulness by looking at internals, not just behaviorally 1/3

Sad: Can't go to CoLM because of immigration. Happy: Well, at least I can mountain bike during the fall break in prime SLC MTB weather. Sad: Comes down with a cold. ☹️☹️☹️☹️☹️☹️

I had a great time chatting with Jekaterina and Malikeh. This episode is like a tour of all the things I've been studying lately!

🎙️ New Women in AI Research episode out now! This time, we sit down with @anamarasovic.bsky.social to unpack some of the toughest questions in AI explainability and trust. 🔗 Watch here → youtu.be/xYb6uokKKOo

Happening today! #COLM2025

In #Interplay25 workshop, Friday ~11:30, I'll present on measuring *parametric* CoT faithfulness on behalf of @mtutek.bsky.social , who couldn't travel: bsky.app/profile/mtut... Later that day we'll have a poster on predicting success of model editing by Yanay Soker, who also couldn't travel

I won't be at COLM, so come see Yonatan talk about our work on estimating CoT faithfulness using machine unlearning! Check out the thread for the (many) other interesting works from his group 🎉

Honored 🎷🎸🥁 𝗠𝗶𝘅𝗔𝘀𝘀𝗶𝘀𝘁 🎷🎸🥁 was selected as the #COLM2025 oral spotlight. Go check out @mclemcrew.bsky.social's 𝐭𝐚𝐥𝐤 on 𝐖𝐞𝐝 (𝐎𝐜𝐭 𝟖) at 𝟑:𝟑𝟎𝐩𝐦 in 517BC and 𝐩𝐨𝐬𝐭𝐞𝐫 from 𝟒:𝟑𝟎-𝟓:𝟑𝟎 in 710!

Mega stoked to attend #COLM25 this week and present our work, MixAssist, on Wednesday! @anamarasovic.bsky.social sadly can't make it 😭, but hit me up if you'd like to chat about audio language models, music mixing, or anything else regarding music and audio!

Our new guest at #WiAIRpodcast is @anamarasovic.bsky.social (Asst prof @ University of Utah , Ex @ Allen AI). We'll talk with her about faithfulness, trust and robustness in AI. The episode is coming soon, don't miss: www.youtube.com/@WomeninAIRe... #WiAIR #NLProc

Pls help spread the word: We are hiring! 🌲☕✨ @ischool.uw.edu is recruiting two tenure-track assistant profs in AI. Areas include human-AI collaboration, AI & society, comp social science, NLP, and AI applications. Apply by Nov 15, 2025: apply.interfolio.com/171020 Happy to answer questions!

Beyond excited to share that I’ll be joining Universal Audio as an LLM Applications Engineer! So stoked to continue working on co-creative AI elements of music production in this role 🎹 S/O to @anamarasovic.bsky.social for teaching me all things NLP these past few years!

Learned a ton working on this survey. Hope it's helpful for you too, check it out! Feedback welcome 🙏 kennethmarino.com/computeruse/...

Super excited that the Computer Use survey I've been working on w/ @anamarasovic.bsky.social for a while now is ready! Originally we were planning on a more traditional survey paper but as more surveys came out we decided on an interactive website survey.

Check @mtutek.bsky.social's thread!

Thrilled that FUR was accepted to @emnlpmeeting.bsky.social Main🎉 In case you can’t wait so long to hear about it in person, it will also be presented as an oral at @interplay-workshop.bsky.social @colmweb.org 🥳 FUR is a parametric test assessing whether CoTs faithfully verbalize latent reasoning.

Welp #acl2025 #acl2025nlp

Now in hall X4, board 407! #ACL2025 #ACL2025NLP

A delayed flight had another plan for my evening 😩

Arriving to #ACL2025 #ACL2025NLP in a few hours! See you at the welcome reception & catch me at the poster session on 𝐓𝐮𝐞𝐬𝐝𝐚𝐲 (𝐉𝐮𝐥𝐲 𝟐𝟗) 𝐚𝐭 𝟏𝟎:𝟑𝟎𝐚𝐦, where Jesse will present our work introducing new tasks for supporting legal brief writing: arxiv.org/abs/2506.06619

The box and I are in similar states

My first audio AI paper, thanks to @mclemcrew.bsky.social who introduced me to a whole new world of co-creative music producing! Getting to work with students who bring their unique passion into PhD work is one of the best perks of a professor's job. Check more in the thread👇🏻 Soon at #colm2025

Reunited with bikes in Bishkek 😮‍💨 Off to 10 days of biking in Kyrgyzstan > Croatia > Vienna (ACL) > more Croatia

After all these years of publishing, still surprised by how much the luck of drawing meaningfully engaged reviewers and an AC, matters in peer review

A new contribution to our line of work building useful AI assistance, this time in legal space. Grateful to co-authors for teaching me about how people do their work in this domain. See more in Fateme's thread 👇

Really excited about this! As backstory, Jesse Woo started this project when I taught a ML Datasets class at Columbia. Then we joined up with @anamarasovic.bsky.social and @fatemehc.bsky.social and really kicked it into high gear. Would not have happened without the full team!

1/ 🚨NEW PAPER: "BriefMe: A Legal NLP Benchmark for Assisting with Legal Briefs", accepted to ACL Findings 2025! We introduce the first benchmark specifically designed to help LLMs assist lawyers in writing legal briefs 🧑‍⚖️ 📄 arxiv.org/abs/2506.06619 🗂️ huggingface.co/datasets/jw4...

🎓 Looking for PhD opportunities in #NLProc for a start in Spring 2026? 🗒️ Add your expression of interest to join @copenlu.bsky.social here by 20 July: forms.office.com/e/HZSmgR9nXB Selected candidates will be invited to submit a DARA fellowship application with me: daracademy.dk/fellowship/f...

And this is why I dread serving as a (S)AC again

Two minutes in my neighborhood walk when ✨✨✨

I need CoLM workshops to extend their deadlines, tyu

ARR reviews from fellow reviewers for all papers assigned to me seem of reasonable quality 😲

Despite your hope, "this email" rarely finds me "well".

If you're going to reuse a subtitle from an existing paper in the area, especially a well-known one, why even bother with the colon-separated title? It's not a requirement

Use "#-MAIN-title" or "#-FIND-title"

But the author response period is similarly quiet 🦗🦗🦗

Does anyone else get an error about paper number and title when registering for ACL? What to do? 😭