Are you wondering if LLM interpretability results generalize, reproduce, etc.? Check out the reproducibility challenge and submit your work reproducing papers in this area: bsky.app/profile/blac...

We don’t always know what problems are hard for LLMs. So devs evaluate on tasks HUMANS find hard or on broad benchmarks. What if we could instead anticipate which scenarios a model will fail on—all without evaluating specific input examples? 🧵NEW PAPER by @jenniferlumeng.bsky.social

Thanks WiAIR (@wiair.bsky.social‬) for featuring my work on your YouTube channel. Watch the video to hear about our work on inference-time steering — and why these interventions LLMs may not be as “precise” as they look.

This call is still open. I am looking to recruit, as well as many other faculty at Cornell. We review folders as they come, and will send offers until all positions are filled. Please share with your network 🙏

What can cognitive science learn from AI? In infinitefaculty.substack.com/p/what-cogni... I outline how AI has found that scale and richness of learning experiences fundamentally change learning & generalization — and how I believe we should rethink cognitive experiments & theories in response.

Woah, this is so cool! How was I not aware of this. I just set mine up to prepare for NeurIPS and I am loving it already... it made thousands of accepted paper so much more tractable to navigate

AIM's 2nd round of TTK hiring - building up to 30 - is up! 📅 Ddl 12/22/25 🔬 Accessibility & Learning, plus Sustainability & Social Justice 🧑‍🏫 Associate/Full Prof* 🔗 umd.wd1.myworkdayjobs.com/en-US/UMCP/j... *Assistant-level candidates: apply to departments, mentioning AIM in a cover letter

My lab at BU is recruiting PhD students and possibly a postdoc this year! We study humans & machines, centered around topics like meaning, generalization, evaluation methods and design, and the nature of computation and representation that underlie language and cognition. 🫴🫴

Interested in interpretability, data attribution, evaluation, and similar topics? Interested in doing a postdoc with me? Apply to the prestigious Azrieli program! Link below 👇 DMs are open (email is good too!)

Happy to be at #EMNLP2025! Please say hello and come see our lovely work

I am recruiting PhD students to start in 2026! If you are interested in robustness, training dynamics, interpretability for scientific understanding, or the science of LLM analysis you should apply. BU is building a huge LLM analysis/interp group and you’ll be joining at the ground floor.

I'll be presenting this work with @rachelrudinger at #NAACL2025 tomorrow (Wednesday 4/30) in Albuquerque during Session C (Oral/Poster 2) at 2pm! 🔬 Decomposing hypotheses in traditional NLI and defeasible NLI helps us measure various forms of consistency of LLMs. Come join us!

What does it mean for #LLM output to be novel? In work w/ johnchen6.bsky.social, Jane Pan, Valerie Chen and He He, we argue it needs to be both original and high quality. While prompting tricks trade one for the other, better models (scaling/post-training) can shift the novelty frontier 🧵

🚨 New Paper 🚨 1/ We often assume that well-written text is easier to translate ✏️ But can #LLMs automatically rewrite inputs to improve machine translation? 🌍 Here’s what we found 🧵

🔈 NEW PAPER 🔈 Excited to share my paper that analyzes the effect of cross-lingual alignment on multilingual performance Paper: arxiv.org/abs/2504.09378 🧵

Have work on the actionable impact of interpretability findings? Consider submitting to our Actionable Interpretability workshop at ICML! See below for more info. Website: actionable-interpretability.github.io Deadline: May 9

Thinking about paying $20k/month for a "PhD-level AI agent"? You might want to wait until their web browsing skills are on par with those of human PhD students 😛 Check out our new BEARCUBS benchmark, which shows web agents struggle to perform simple multimodal browsing tasks!

🚨 Our team at UMD is looking for participants to study how #LLM agent plans can help you answer complex questions 💰 $1 per question 🏆 Top-3 fastest + most accurate win $50 ⏳ Questions take ~3 min => $20/hr+ Click here to sign up (please join, reposts appreciated 🙏): preferences.umiacs.umd.edu

🚨 New Position Paper 🚨 Multiple choice evals for LLMs are simple and popular, but we know they are awful 😬 We complain they're full of errors, saturated, and test nothing meaningful, so why do we still use them? 🫠 Here's why MCQA evals are broken, and how to fix them 🧵

How can we generate synthetic data for a task that requires global reasoning over a long context (e.g., verifying claims about a book)? LLMs aren't good at *solving* such tasks, let alone generating data for them. Check out our paper for a compression-based solution!

This paper is really cool. They decompose NLI (and defeasible NLI) hypotheses into atoms, and then use these atoms to measure the logical consistency of LLMs. E.g. for an entailment NLI example, each hypothesis atom should also be entailed by the premise. Very nice idea 👏👏

Please join us for: AI at Work: Building and Evaluating Trust Presented by our Trustworthy AI in Law & Society (TRIALS) institute. Feb 3-4 Washington DC Open to all! Details and registration at: trails.gwu.edu/trailscon-2025 Sponsorship details at: trails.gwu.edu/media/556

The Impact of Explanations on Fairness in Human-AI Decision-Making: Protected vs Proxy Features Despite hopes that explanations improve fairness, we see that when biases are hidden behind proxy features, explanations may not help. Navita Goyal, Connor Baumler +al IUI’24 hal3.name/docs/daume23... >

This is my first time serving as an AC for a big conference. Just read this great work by Goyal et al. arxiv.org/abs/2411.11437 I'm optimizing for high coverage and low redundancy—assigning reviewers based on relevant topics or affinity scores alone feels off. Seniority and diversity matter!

Large Language Models Help Humans Verify Truthfulness—Except When They Are Convincingly Wrong Should one use chatbots or web search to fact check? Chatbots help more on avg, but people uncritically accept their suggestions much more often. by Chenglei Si +al NAACL’24 hal3.name/docs/daume24... >