Can AI models introspect? What does introspection even mean for AI? We revisit a recent proposal by Comșa & Shanahan, and provide new experiments + an alternate definition of introspection. Check out this new work w/ @siyuansong.bsky.social, @harveylederman.bsky.social, & @kmahowald.bsky.social 👇

Due to popular demand, we are extending the CogInterp submission deadline again! 🗓️🥳 Submit by *8/27* (midnight AoE)

🗓️ The submission deadline for CogInterp @ NeurIPS has officially been *extended* to 8/22 (AoE)! 👇 Looking forward to seeing your submissions!

Heading to CogSci this week! ✈️ Find me giving talks on: 💬 Prod-comp asymmetry in children and LMs (Thu 7/31) 💬 How people make sense of nonsense (Sat 8/2) 📣 Also, I’m recruiting grad students + postdocs for my new lab at Hopkins! 📣 If you’re interested in language / cognition / AI, let’s chat! 😄

Excited to announce the first workshop on CogInterp: Interpreting Cognition in Deep Learning Models @ NeurIPS 2025! 📣 How can we interpret the algorithms and representations underlying complex behavior in deep learning models? 🌐 coginterp.github.io/neurips2025/ 1/4

Happy to announce the first workshop on Pragmatic Reasoning in Language Models — PragLM @ COLM 2025! 🎉 How do LLMs engage in pragmatic reasoning, and what core pragmatic capacities remain beyond their reach? 🌐 sites.google.com/berkeley.edu/praglm/ 📅 Submit by June 23rd

Excited to share a new preprint w/ @michael-lepori.bsky.social & Michael Franke! A dominant approach in AI/cogsci uses *outputs* from AI models (eg logprobs) to predict human behavior. But how does model *processing* (across layers in a forward pass) relate to human real-time processing? 👇 (1/12)

Check out our new work on introspection in LLMs! 🔍 TL;DR we find no evidence that LLMs have privileged access to their own knowledge. Beyond the study of LLM introspection, our findings inform an ongoing debate in linguistics research: prompting (eg grammaticality judgments) =/= prob measurement!

new preprint on Theory of Mind in LLMs, a topic I know a lot of people care about (I care. I'm part of people): "Re-evaluating Theory of Mind evaluation in large language models" (by Hu* @jennhu.bsky.social , Sosa, and me) link: arxiv.org/pdf/2502.21098

AI models are fascinating, impressive, and sometimes problematic. But what can they tell us about the human mind? In a new review paper, @noahdgoodman.bsky.social and I discuss how modern AI can be used for cognitive modeling: osf.io/preprints/ps...

Some things are more impossible than others. But some things might be even *more impossible* than impossible. (How) do people differentiate between the inconceivable and the merely impossible? Do language models also make similar distinctions? Check out our new preprint below!

Hello! I'm looking to hire a post-doc, to start this Summer or Fall. It'd be great if you could share this widely with people you think might be interested. More details on the position & how to apply: bit.ly/cocodev_post... Official posting here: academicpositions.harvard.edu/postings/14723

Now hiring for two lab manager positions at Stanford! Hyo Gweon and I are coordinating joint searches since our labs collaborate frequently. Please join us! careersearch.stanford.edu/jobs/researc... and careersearch.stanford.edu/jobs/lab-coo...

(1/9) Excited to share my recent work on "Alignment reduces LM's conceptual diversity" with @tomerullman.bsky.social and @jennhu.bsky.social, to appear at #NAACL2025! 🐟 We want models that match our values...but could this hurt their diversity of thought? Preprint: arxiv.org/abs/2411.04427

LMs need linguistics! New paper, with @futrell.bsky.social, on LMs and linguistics that conveys our excitement about what the present moment means for linguistics and what linguistics can do for LMs. Paper: arxiv.org/abs/2501.17047. 🧵below.

Stop by our #NeurIPS tutorial on Experimental Design & Analysis for AI Researchers! 📊 neurips.cc/virtual/2024/tutorial/99528 Are you an AI researcher interested in comparing models/methods? Then your conclusions rely on well-designed experiments. We'll cover best practices + case studies. 👇

To researchers doing LLM evaluation: prompting is *not a substitute* for direct probability measurements. Check out the camera-ready version of our work, to appear at EMNLP 2023! (w/ @rplevy.bsky.social) Paper: arxiv.org/abs/2305.13264 Original thread: twitter.com/_jennhu/stat...