Thoughtful (as always) blog post from Nicholas Carlini. "Are large language models worth it?" A nice read giving his perspective on risks of ML models. Post: nicholas.carlini.com/writing/2025... For people who prefer, this is the video of the talk from @colmweb.org www.youtube.com/watch?v=PngH...

Extremely thrilled to talk about our new paper: "Who Evaluates AI’s Social Impacts? Mapping Coverage And Gaps In First And Third Party Evaluations". This is the first big project output from the @eval-eval.bsky.social coalition! Thread below:

Congratulations @sivareddyg.bsky.social ! 🥳 Incredibly well deserved!!

We're excited to announce that the website and registration for IC2S2 2026 (July 28-31) will launch in early December! The Vermont Complex Systems Institute @vcsi.bsky.social at the University of Vermont will be hosting IC2S2 in 2026: youtube.com/watch?v=p412S4GnPkc&feature=youtu.be

It's the season for PhD apps!! 🥧 🦃 ☃️ ❄️ Apply to Wisconsin CS to research - Societal impact of AI - NLP ←→ CSS and cultural analytics - Computational sociolinguistics - Human-AI interaction - Culturally competent and inclusive NLP with me! lucy3.github.io/prospective-...

It’s grad school application season, and I wanted to give some public advice. Caveats: -*-*-*-* 
> These are my opinions, based on my experiences, they are not secret tricks or guarantees 
> They are general guidelines, not meant to cover a host of idiosyncrasies and special cases

🧵Excited to present our work at #EMNLP2025 “Analyzing Dialectal Biases in LLMs for Knowledge and Reasoning Benchmarks”! Paper 📄 arxiv.org/abs/2510.00962 w/ Eileen Pan, Skyler Seto, @allisonkoe.bsky.social @maartjeterhoeve.bsky.social

it's crazy how typos are impossible to catch until *after* you submit a paper, after which they become glaringly noticeable

Which, whose, and how much knowledge do LLMs represent? I'm excited to share our preprint answering these questions: "Epistemic Diversity and Knowledge Collapse in Large Language Models" 📄Paper: arxiv.org/pdf/2510.04226 💻Code: github.com/dwright37/ll... 1/10

What makes dialogue 💬 constructive 🫂? We address this question in our #EMNLP2025 paper investigating how **responsivity** can characterize conversation quality ✨ Brandon Roy will be presenting our work (Oral) on Nov 7, Room A109 at 10:30. 🧵👇 ​​https://aclanthology.org/2025.emnlp-main.1798/

Great piece by @natolambert.bsky.social on the current state of human exhaustion in the AI world. Makes this important point:

👀 www.bbc.co.uk/mediacentre/...

This paper underscores how “misinformation” is not easily operationalized as single pieces of factually untrue content, but often takes shape through the motivated amplification of specific evidence (often true evidence) that serves to build and reinforce misleading frames.

Really proud bc I finished my first crochet sweater! It’s pretty cool wearing something you made by hand, every single stitch, mistakes and all

MIT rejects "compact" proposed by the Trump administration. MIT prez wrote: it "would restrict freedom of expression and our independence as an institution" and "is inconsistent with our core belief that scientific funding should be based on scientific merit alone." orgchart.mit.edu/letters/rega...

Hello #COLM2025! Excited to be kicking off the NLP4Democracy workshop this morning. We are in 520E (behind A/B/C) - check out our amazing program! sites.google.com/andrew.cmu.e...

💡We kicked off the SoLaR workshop at #COLM2025 with a great opinion talk by @michelleding.bsky.social & Jo Gasior Kavishe (joint work with @victorojewale.bsky.social and @geomblog.bsky.social ) on "Testing LLMs in a sandbox isn't responsible. Focusing on community use and needs is."

Hi #COLM2025! 🇨🇦 I will be presenting a talk on the importance of community-driven LLM evaluations based on an opinion abstract I wrote with Jo Kavishe, @victorojewale.bsky.social and @geomblog.bsky.social tomorrow at 9:30am in 524b for solar-colm.github.io Hope to see you there!

Important keynote by Nicholas Carlini with important calls to action for the research community! Ty for the helpful summary @mariaa.bsky.social

COLM word cloud. Yoav says it’s the year of reasoning, but evaluation is also huge.

I’m at #COLM2025! Would love to chat about anything related to pluralistic alignment, fairness evaluations, societal impacts of LLMs, etc 😊 You can also find me at the NLP4Democracy workshop giving a talk about my work analyzing democratic deliberation with LLMs Oct 10th!

Alright the evening sky, you’re utterly wondrous and fantastical, we get it, geez

Here’s a #COLM2025 feed! Pin it 📌 to follow along with the conference this week!

I will be at #COLM2025 this week, and would love to connect with folks interested in applications (and critiques) of language modeling in social science research! And join us for the NLP4Democracy workshop on Friday! sites.google.com/andrew.cmu.e... #NLP #NLProc #LLM #ComputationalSocialScience

Very excited that my paper with @katakeith.bsky.social is now out in @polanalysis.bsky.social. We investigate whether LLMs actually follow the instructions/definitions provided in codebooks, propose some diagnostics, and release a new evaluation dataset. www.cambridge.org/core/journal...

I wish students understood in most empirical AI research there’s a huge scientific advantage from being constitutionally excited by math vs intimidated, but very little additional gain from being actually “good” at math. Maybe they’d be less intimidated if they didn’t feel they had to be “good”.

🚨 New preprint! 🚨 Excited to share my work: An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies 🤖🗳️ I’ll be presenting this at @colmweb.org in the NLP4Democracy workshop! 🔗 arxiv.org/abs/2509.12577

"This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity." super interesting

This paper yields the same conclusion as what @mustafasuleymanai.bsky.social recently posted on the danger of 'seemingly conscious AI'. mustafa-suleyman.ai/seemingly-co...

When reading papers, especially reviewing, I like to print and annotate as I read. I wish I could upload this to open review so authors can see smaller suggestions (typos, formatting errors) as well as smaller positive notes eg things I appreciated or found useful/interesting

First crochet project done! Super proud and excited for the next one

This new center strikes the right tone in approaching the AI alignment problem. alignmentalignment.ai

It never ceases to amaze me that some folks can just continue on with their evenings and not stop everything to gaze at the magnificence of the post-rain sunset (and ofc take a billion photos)!

This is a cool paper that suggests that AI agents can indeed be used for social science experiments, but that just using a chatbot isn't good enough, instead prompts developed based on social & game theory makes AI agent actions predictive of real human outcomes. benjaminmanning.io/files/optimi...

It’s cool that Massachusetts is getting a better flag but disappointing that we weren’t brave enough to select this entry from the competition

I had an intense summer so I’m recovering with a vacork. It’s the opposite of a workation: I go to the office but I don’t get anything done, just drink coffee and distract other people

I got the complaint in the horrific OpenAI self harm case the the NY Times reported today This is way way worse even than the NYT article makes it out to be OpenAI absolutely deserves to be run out of business

Adam Raine, 16, died from suicide in April after months on ChatGPT discussing plans to end his life. His parents have filed the first known case against OpenAI for wrongful death. Overwhelming at times to work on this story, but here it is. My latest on AI chatbots: www.nytimes.com/2025/08/26/t...

Our work on Detecting Suspense in Stories will appear at COLM'25 arxiv.org/abs/2508.15794 Do LLMs know when stories are suspenseful? We ran LLMs through a bunch of classical psychology suspense studies The answer: kinda, sorta? Honestly, better than I expected. But the results are nuanced... 1/

I was reflecting on times i felt / pursued real curiosity, and decided to write a small post on the time I decided to learn category theory to study applications to language model + interpretability in 2022 with the mentorship of Prakash Panangaden elinorp-d.github.io/projects/cat...

Congratulations @bennokrojer.bsky.social on passing your PhD proposal exam! A great presentation and exciting work!

A nice discovery today: #ic2s2 keynotes are on Youtube! In particular, I think the papers listed by @lauraknelson.bsky.social in her keynote could form a great list for a CSS reading group. www.youtube.com/watch?v=NXYg...

Our new paper in #PNAS (bit.ly/4fcWfma) presents a surprising finding—when words change meaning, older speakers rapidly adopt the new usage; inter-generational differences are often minor. w/ Michelle Yang, ‪@sivareddyg.bsky.social‬ , @msonderegger.bsky.social‬ and @dallascard.bsky.social‬👇(1/12)

The Copenhagen airport (CPH) has amazing plants. The large mature flowering Hoya Australis’ are so stunning!

Talking about this work tomorrow (Wed, July 23rd) at #IC2S2 in Norrköping during the 11 am session on LLMs, Annotation, and Synthetic Data! Come hear about this and more!

Hope Schroeder, Deb Roy, Jad Kabbara: Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks https://arxiv.org/abs/2507.15821 https://arxiv.org/pdf/2507.15821 https://arxiv.org/html/2507.15821

Not sure who needs to hear this but what people want AI systems to do, what AI systems do, and what people believe AI systems do are not the same thing. Just because one wants or believes AI systems do or can do certain things, doesn't mean they actually do those things.

Introducing: Full-Stack Alignment 🥞 A research program dedicated to co-aligning AI systems *and* institutions with what people value. It's the most ambitious project I've ever undertaken. Here's what we're doing: 🧵

So, full-stack alignment is our way of saying that we need AI and institutions that "fit" us, and that help us live the lives we want to live.