One more CoLM meta-review to write, yay... ...the last paper's reviews+discussion contain about **16,000 words** and the paper is a borderline, RIP

How uninformed you have to be to suggest this prompt... “Create a research grant proposal focusing on [topic]. Align it with [funding agency] guidelines (linked) and ensure the proposal includes a robust literature review, detailed methods section [...]” academy.openai.com/public/clubs...

Why is 4 still the minimum number of papers one can sign up to review for ARR??

Analog camera + desert 🧡

I am still not caught up

Is it just me or Claude chat interface is really buggy these days?

For a recent lab meeting, I wrote up a grab bag of ways to think about your development as a researcher during a PhD: emerge-lab.github.io/papers/an-un... Sharing in case folks find it useful or have feedback!

Been less than a year since I started my lab at @utah.edu and we already have a ton of new stuff that I can’t wait to talk about soon. I’ll start today by sharing that our updated Computer Use Survey blog has been accepted to ICLR Blogposts 2026. iclr-blogposts.github.io/2026/blog/20...

Great test for anyone learning mech interp is reading Nikankin et al's "Arithmetic Without Algorithms" which uses activation patching / circuits, probing, logit lens, describing max activating examples.. If you you follow along while reading, you'll realize you know a lot! arxiv.org/abs/2410.21272

Getting sick in the middle of the semester, like catching a flu, makes every next week of the semester progressively worse. Will I ever catch up 😭

I know grant writing is supposed to be miserable. Maybe once I've written a few and had them all rejected I will agree. But it's kind of my favorite genre to write currently? It's pure optimism! These are my hopes for the future! My favorite ideas! Untainted by messy and limited results!

Somehow before becoming a prof I came away with the impression that grant writing was an annoying task profs have to do, and yes, more rejection sucks, but it is wonderful to start new collaborations with super smart people, brainstorm hard, and think on a larger scale than next few papers

It's so depressing to see 20-30K papers submitted to every top ML conference

Mathematicians are really nice to their LLMs; seen in: arxiv.org/abs/2601.01235

New year, new porcupine sighting!

We need to return word limit to author responses

We are hiring in AI and NLP at Utah! Please apply if interested and share with folks you know are looking for faculty positions, especially if they like to ski, hike, climb, or bike! utah.peopleadmin.com/postings/190...

💫 On the heels of announcing 12 new faculty fellows last week, SCI's One-U Responsible AI Initiative is excited to add three new postdoctoral fellows to its team next year: rai.utah.edu/postdocs-dec...

KSL TV @ksl.com interviewed the members of my research group who recently discovered a rare piece of computing history: an old tape that might contain UNIX V4. ksltv.com/science-tech...

Make it make sense 😂

📣 Meet the 12 new faculty fellows joining SCI’s One-U Responsible AI Initiative. These professors are advancing AI research to solve real-world challenges, from protecting the West’s water supply to improving medical care to embedding ethics in AI education. bit.ly/rai-fac-26

Is it me or more reviewers are -strongly- opinionated about how you paper should be written? I feel like my responses in the past year or two are turning into arguments about why my writing is just fine

Current ChatGPT response format is such an eyesore, what’s up with all the lists and emojis

Luca Guadagnino making an OpenAI movie, what's happening 👀👀

I didn't submit to ICLR, but I'm pretty sure next week I'll see similar quality issues in ARR I feel like execs of major ML/AI conferences from ICLR/NeurIPS/ICML/AAAI, ACL/EMNLP, to CVPR should sit together and figure out a whole new strategy moving forward like 👇

When your husband is also an academic, so when you can't get him on a phone you shoot an email and it works every time 😂💀

We're surveying researchers about name changes in academic publishing. If you've changed your name and dealt with updating publications, we want to hear your experience. Any reason counts: transition, marriage, cultural reasons, etc. forms.cloud.microsoft/e/E0XXBmZdEP

@mclemcrew.bsky.social's CoLM spotlight is now available on YT! 🎵 youtu.be/w6LNmADnlNw?...

*Urgently* looking for emergency reviewers for the ARR October Interpretability track 🙏🙏 ReSkies much appreciated

Outstanding paper (5/7): "Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps" by Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, and Yonatan Belinkov aclanthology.org/2025.emnlp-m... 6/n

𝙒𝙚'𝙧𝙚 𝙝𝙞𝙧𝙞𝙣𝙜 𝙣𝙚𝙬 𝙛𝙖𝙘𝙪𝙡𝙩𝙮 𝙢𝙚𝙢𝙗𝙚𝙧𝙨! KSoC: utah.peopleadmin.com/postings/190... (AI broadly) Education + AI: - utah.peopleadmin.com/postings/189... - utah.peopleadmin.com/postings/190... Computer Vision: - utah.peopleadmin.com/postings/183...

🎉 Congratulations to all #EMNLP2025 award winners 🎉 Starting with the ✨Best Paper award ✨: "Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index" by Hao Xu, Jiacheng Liu, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi aclanthology.org/2025.emnlp-m... 1/n

Thrilled to see this work recognized at #EMNLP2025! This framework and approach to measuring CoT faithfulness have been hugely influential for how I think about reasoning evaluation, and I'm so lucky to have worked with such brilliant collaborators. Huge credit to @mtutek.bsky.social

Very honored to be one out of seven outstanding papers at this years' EMNLP :) Huge thanks to my amazing collaborators @fatemehc.bsky.social @anamarasovic.bsky.social @boknilev.bsky.social , this would not have been possible without them!

Check out Martin's talk at #EMNLP2025 today (Wed)! If you care about CoT faithfulness, you 𝘮𝘶𝘴𝘵 read this paper. It introduces the first method for measuring CoT faithfulness that is not purely behavioral, but operates with the internals!

Go check Alex's poster today (Wed) in Suzhou! #EMNLP2025 I'm still so proud of our work (led by @lasha.bsky.social) on CondaQA, so we had to ask what would happen if we tried to create high-quality reasoning-over-text benchmarks now that LLMs are available. Turns out, we'd make an easier benchmark!

Flying out to @emnlpmeeting soon🇨🇳 I'll present our parametric CoT faithfulness work (arxiv.org/abs/2502.14829) on Wednesday at the second Interpretability session, 16:30-18:00 local time A104-105 If you're in Suzhou, reach out to talk all things reasoning :)

Sort of starting to believe that we really do need academic metrics that punish publishing too much

I'll be in Suzhou 🇨🇳 at #EMNLP this week presenting "What has been Lost with Synthetic Evaluation?" done with @anamarasovic.bsky.social & @lasha.bsky.social! 🎉 📍Findings Session 1 - Hall C 📅 Wed, November 5, 13:00 - 14:00 arxiv.org/abs/2505.22830

🧠 Can large language models build the very benchmarks used to evaluate them? In “What Has Been Lost with Synthetic Evaluation”, Ana Marasović (@anamarasovic.bsky.social) and collaborators ask what happens when LLMs start generating the datasets used to test their reasoning. (1/6🧵)

👉 Do large language models really reason the way their chain-of-thoughts suggest? This week on #WiAIRpodcast, we talk with Ana Marasović (@anamarasovic.bsky.social) about her paper “Chain-of-Thought Unfaithfulness as Disguised Accuracy.” (1/6🧵) 📄 Paper: arxiv.org/pdf/2402.14897

Can you trust your reward model alignment scores? New work presented today at the COLM Workshop on Socially Responsible Language Modelling Research led by Purbid Bambroo and in collaboration with @anamarasovic.bsky.social that probes LLM preference test sets for redundancy and inflated scores. 1/8

📣Tomorrow at #COLM2025: 1️⃣ Purbid's 𝐩𝐨𝐬𝐭𝐞𝐫 at 𝐒𝐨𝐋𝐚𝐑 (𝟏𝟏:𝟏𝟓𝐚𝐦-𝟏:𝟎𝟎𝐩𝐦) on catching redundant preference pairs & how pruning them hurts accuracy; www.anamarasovic.com/publications... 2️⃣ My 𝐭𝐚𝐥𝐤 at 𝐗𝐋𝐋𝐌-𝐑𝐞𝐚𝐬𝐨𝐧-𝐏𝐥𝐚𝐧 (𝟏𝟐𝐩𝐦) on measuring CoT faithfulness by looking at internals, not just behaviorally 1/3

📣Tomorrow at #COLM2025: 1️⃣ Purbid's 𝐩𝐨𝐬𝐭𝐞𝐫 at 𝐒𝐨𝐋𝐚𝐑 (𝟏𝟏:𝟏𝟓𝐚𝐦-𝟏:𝟎𝟎𝐩𝐦) on catching redundant preference pairs & how pruning them hurts accuracy; www.anamarasovic.com/publications... 2️⃣ My 𝐭𝐚𝐥𝐤 at 𝐗𝐋𝐋𝐌-𝐑𝐞𝐚𝐬𝐨𝐧-𝐏𝐥𝐚𝐧 (𝟏𝟐𝐩𝐦) on measuring CoT faithfulness by looking at internals, not just behaviorally 1/3

Sad: Can't go to CoLM because of immigration. Happy: Well, at least I can mountain bike during the fall break in prime SLC MTB weather. Sad: Comes down with a cold. ☹️☹️☹️☹️☹️☹️

I had a great time chatting with Jekaterina and Malikeh. This episode is like a tour of all the things I've been studying lately!

🎙️ New Women in AI Research episode out now! This time, we sit down with @anamarasovic.bsky.social to unpack some of the toughest questions in AI explainability and trust. 🔗 Watch here → youtu.be/xYb6uokKKOo

Happening today! #COLM2025

In #Interplay25 workshop, Friday ~11:30, I'll present on measuring *parametric* CoT faithfulness on behalf of @mtutek.bsky.social , who couldn't travel: bsky.app/profile/mtut... Later that day we'll have a poster on predicting success of model editing by Yanay Soker, who also couldn't travel

I won't be at COLM, so come see Yonatan talk about our work on estimating CoT faithfulness using machine unlearning! Check out the thread for the (many) other interesting works from his group 🎉