Most Multilingual benchmarks measure what models know, not what they can reliably do. Our #ACL2026 paper introduces functional benchmarks in six languages from English to Yoruba to test whether models can actually execute tasks across languages, not just answer fixed questions about them. 🧵1/n

Attending the @eval-eval.bsky.social workshop at #ACL2026 to present this paper! Happy to chat about all things LLM-as-a-judge + the role of human feedback/participation in designing evals ✨

it's the last day of #FAccT2026 in Montreal (bonjour hiii), and I'm presenting my paper with Diag Davenport on the promise of #PublicInterestTech clinics for training the next gen of critical sociotechnical thinkers. 🏆 plus we got an honorable mention!? come thru @ 10:45! 📄 tiny.cc/pit-clinics-26

Super excited to share a new #FAccT2026 paper with Nel Escher and Nikola Banovic! We show that algorithm auditing policies in the US are wildly insufficient, as they don't account for how public sector algorithms actually function in practice. dl.acm.org/doi/abs/10.1...

RLHF is, instead, a form of survey, which is sensitive to the framing of the solicitation questions and other survey design practices. It's also a form of inverted content moderation - sometimes described as "content moderation via vibes". This moderation operates over generation, not screening.

We want human preferences to be an objective measure of human values, but this view is not grounded in RLHF practice. Instead, the raters who provide preferences are screened to express certain values and are then trained to provide preferences that adhere to some specific criteria. #FAccT2026

All AI evaluations embed assumptions about what "good" behavior looks like. Our #FAccT2026 paper explores how we can center the perspectives of impacted communities - specifically, subjects of AI-generated media - in designing LLM-as-a-judge evaluation rubrics.

yessss! People often don't realize that policy change is often a 2-3 year game of getting stuff drafted and socialized and "waiting in the wings" for the right moment. NOW IS THE TIME

Policymakers by and large don't read academic papers! If you're looking for policy impact, you need to show up. First you need to learn _how_ to show up. Excited to cohost this #FAccT2026 craft session with @rone.bsky.social, @narijohnson.bsky.social, Cynthia Bailey, and Danaé Metaxa.

🔬Science can play a critical role in guiding policy interventions. How can we make our research legible to the policymakers who work on AI? Our tutorial "From FAccT to Policy Impact" explores how we can translate our research in a specific format: the policy brief! #FAccT2026

AI companies are paying screenwriters, lawyers, and other white-collar professionals to produce the training data needed to automate their jobs. I spoke with more than 30 workers about conditions inside this fast-growing and extremely secretive new gig economy.

🎨💻 What is a “high-quality” or “aesthetic” image according to generative AI developers? Happy to share that our investigation of the LAION-Aesthetics Predictor has been accepted at #FAccT2026! 🧵 (1/5) Take a look at a preprint here: arxiv.org/abs/2601.09896

🚨The 2025 AI Agent Index is out! 🚨 Amidst recent buzz over 🦀 and NIST's new agent initiative, we find: - Selective reporting – esp. on safety - Almost all agents backend just 3 model families - Many agents don’t ID themselves as bots online - Big US/China gaps - And more…

We opened up applications for the Brown AI Policy Summer School! Please share with any computing or computational social science students who want to engage substantively with policymaking in the United States: cntr.brown.edu/summer-school. Deadline March 27th!!! Funding available!

The deadline for the 2026 FAccT DC is next Tuesday, February 24! If you are a student working on topics relevant to the FAccT's scope, this is an opportunity to interact with a diverse set of peers and mentors! #facct2026 #facct26 #facct Details here: facctconference.org/2026/callfor...

🔮 How can we empower online communities to design AI agents tailored to their unique needs and norms? In our #CHI2026 paper, we introduce #Botender, a system that enables collaborative design of AI agents through 🔥case-based provocation🔥

PhD admissions visits/open houses are starting to happen, and I got a comment on an old Reddit post where I was offering advice, and realized that it's actually really good advice. So here it is! (And this applies whether you've already been admitted to the program or not.) 🧵

Yep and it gets worse! Owner doesn't even care to remove hundreds of skills which directly instruct the model to install malware opensourcemalware.com/blog/clawdbo...

Our call for craft and tutorial sessions for #FAccT2026 is now live! ▶️ Craft CfP: facctconference.org/2026/cfpcraf... ▶️ Tutorials CfP: facctconference.org/2026/cft.html Both kinds of proposals are due March 25!

🎭 How do LLMs (mis)represent culture? 🧮 How often? 🧠 Misrepresentations = missing knowledge? spoiler: NO! At #CHI2026 we are bringing ✨TALES✨ a participatory evaluation of cultural (mis)reps & knowledge in multilingual LLM-stories for India 📜 arxiv.org/abs/2511.21322 1/10

Microsoft Research NYC is hiring a researcher in the space of AI and society!

A new report by the Center for Tech Responsibility at Brown University and the ACLU uses computational tools to analyze legislative trends on AI across 1,804 state and federal bills, while offering recommendations for how to integrate the technology into policy analysis.

We are studying the sentiments of visual artists towards generative AI in the workplace and their impacts on creative careers. If you're an artist, please consider filling out this recruitment form for access to our survey! cmu.ca1.qualtrics.com/jfe/form/SV_...

Most LLM evals use API calls or offline inference, testing models in a memory-less silo. Our new Patterns paper shows this misses how LLMs actually behave in real user interfaces, where personalization and interaction history shape responses: arxiv.org/abs/2509.19364

US CAISI is hiring -- the internal govt name for the role is "IT Specialist" but it is effectively a research scientist role! Salary is $120,579 to - $195,200 per year, and you get to work on AI evaluation within government agencies! Job posting (**closes EOD 12/28/2025**): lnkd.in/exJgkqr5

Also, our team is hiring an AI Research Scientist! www.usajobs.gov/job/851528400

Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science: www.nist.gov/blogs/caisi-...

Did you know that one base model is responsible for 94% of model-tagged NSFW AI videos on CivitAI? This new paper studies how a small number of models power the non-consensual AI video deepfake ecosystem and why their developers could have predicted and mitigated this.

I appreciate this sympathetic position people's feelings of emotional dependency on these "human-like" bots is real. ridiculing them doesn't help anyone

Can public involvement in AI evaluation improve the science? Or does it compromise quality, speed, cost? In @pnas.org, Megan Price & I summarize challenges of AI evaluation, review strengths/weaknesses, & suggest how participatory methods can improve the science of AI www.pnas.org/doi/10.1073/...

Can LLMs accurately aggregate information over long, information-dense texts? Not yet… We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!

📣 Our method for conducting community-based algorithmic impact assessments is now available! We’ve just launched a new section on our website where you can find an extensive toolkit, documentation of our pilots, and a series of reflections on lessons learned. datasociety.net/research/alg...

𝐒𝐨𝐜𝐢𝐞𝐭𝐚𝐥 𝐈𝐦𝐩𝐚𝐜𝐭 𝐀𝐬𝐬𝐞𝐬𝐬𝐦𝐞𝐧𝐭 𝐟𝐨𝐫 𝐈𝐧𝐝𝐮𝐬𝐭𝐫𝐲 𝐂𝐨𝐦𝐩𝐮𝐭𝐢𝐧𝐠 𝐑𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫𝐬 🏅 Best Paper Honorable Mention (Top 3% Submissions) 🔗 dl.acm.org/doi/10.1145/... 📆 Wed, 22 Oct | 9:00 AM, CET: Toward More Ethical and Transparent Systems and Environments

💡Can we trust synthetic data for statistical inference? We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data

Our Responsible AI team at Apple is looking for spring/summer 2026 PhD research interns! Please apply at jobs.apple.com/en-us/detail... and email [email protected]. Do not send extra info (e.g., CV), just drop us a line so we can find your application in the central pool!

✨I’m on the academic job market ✨ I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻‍💻💪🏽💥 I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action More info: cella.io

🌟 If you’re applying to CMU SCS PhD programs, and come from a background that would bring additional dimensions to the CMU community, our PhD students are here to help! Apply to the Graduate Applicant Support Program by Oct 13 to receive feedback on your application materials:

📌📌📌 I'm excited to be on the faculty job market this fall. I just updated my website with my CV. stephencasper.com

📢2026 Fellowship applications are OPEN!📢 If you are someone looking to inform technology policy through rigorous original reporting or policy analyses, we want to hear from you! Apply here: airtable.com/appIrc1F9M5d...

What can #CSCW learn from tech workers who have been involved in collective action and unionization about how to make transformative change within our field? My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question. Pre-print: arxiv.org/pdf/2508.12579

The exchanges between Adam and ChatGPT are devastating. This, in my mind, is the worst one. One of his last messages was a photo of the noose hung in his bedroom closet, asking if it was "good." ChatGPT offered a technical analysis of the set up and told him it 'could potentially suspend a human."

Adam Raine, 16, died from suicide in April after months on ChatGPT discussing plans to end his life. His parents have filed the first known case against OpenAI for wrongful death. Overwhelming at times to work on this story, but here it is. My latest on AI chatbots: www.nytimes.com/2025/08/26/t...

A cognitively impaired New Jersey man grew infatuated with a Meta chatbot originally created in partnership with celebrity influencer Kendall Jenner. His fatal attraction shines a light on Meta's guidelines for its AI chatbots reut.rs/45DQIRj @jeffhorwitz.bsky.social

🧵 New paper from UK AISI x @eleutherai.bsky.social rai.bsky.social‬ that I led with @kyletokens.bsky.social y.social���: Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.

* STS folks! * CMU is hiring up to 2 tenure track faculty focused on: the intersection of tech & social change, the environmental and social impacts of science, tech, and medicine. They will be housed in History, a department of both historians and anthropologists. apply.interfolio.com/170040

One of the largest text-image datasets is full of PII, including credit card numbers and birth certificates. Excellent writeup by @eileenguo.bsky.social www.technologyreview.com/2025/07/18/1... Read our full audit and legal analysis arxiv.org/pdf/2506.17185

If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...

🏦 Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees’ Practices, Challenges, and Needs by @narijohnson.bsky.social et al. explores procurement in the context of recent calls for governments to use their "purchasing power" to incentivize responsible AI.

I did an interview w/ Pittsburgh's NPR station to share some of my views on the topic of the McCormick/Trump AI & Energy summit at CMU tomorrow. Despite being hosted at the university, there will not be opportunities for our university experts to contribute viewpoints at the event.

New article out today, covering our past two years of research asking US cities how they govern AI ✨ The future of AI governance in public services is being shaped right now, through public procurement