AI benchmark evals commonly report "accuracy" metrics — but what’s really being measured? And how should we compute the error bars? New NIST report from my team at CAISI outlines a better statistical framework for eval analysis: www.nist.gov/news-events/...

This has been a massive community project, and we need you all to participate! See more: evalevalai.com/projects/eve...

Had the chance to give feedback on this project on CAISI’s behalf. I’m very excited to see this develop!

If this kind of work speaks to you, come work with us! My team at CAISI is hiring an Applied Systems AI Research Scientist, among many other roles. www.nist.gov/caisi/career...

Excited to co-author a new public draft from NIST CAISI on best practices for automated benchmark evals. We want your feedback! Public comment open til March 31. Highlights below 🧵 www.nist.gov/news-events/...

Late notice, but: NIST has a multi-year, funded graduate fellowship + summer intern program (potentially with our team at CAISI!). Due January 15. stemfellowships.org/applicants/

Are you passionate about exploring what a conceptually cogent, methodologically sound, and well-founded AI evaluation and safety research might look like? Come do a PhD with us. Closing Date: 10 February 2026 Apply here aial.ie/hiring/phd-a...

US CAISI is hiring -- the internal govt name for the role is "IT Specialist" but it is effectively a research scientist role! Salary is $120,579 to - $195,200 per year, and you get to work on AI evaluation within government agencies! Job posting (**closes EOD 12/28/2025**): lnkd.in/exJgkqr5

Also, our team is hiring an AI Research Scientist! www.usajobs.gov/job/851528400

Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science: www.nist.gov/blogs/caisi-...

US CAISI (the equivalent of the US "AI Safety Institute") just put out their approach to AI measurement & there's such a significant portion on construct validity (nist.gov/blogs/caisi-...). Great to see this after ongoing advocating about this issue (arxiv.org/abs/2511.04703)!

After having such a great time at #CHI2025 and #FAccT2025, I wanted to share some of my favorite recent papers here! I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!! 🧵⬇️:

Q: What do school buses, desktop computers, and AI have in common? A: The same decades-old laws and processes apply when governments go to purchase them. Our new #FAccT2025 paper examines how these legacy public procurement norms apply to AI. 🧵

Excited to present "Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling" at #CHI2025 tomorrow(today)! 🗓 Tue, 29 Apr | 9:48–10:00 AM JST (Mon, 28 Apr | 8:48–9:00 PM ET) 📍 G401 (Pacifico North 4F) 📄 dl.acm.org/doi/10.1145/...

Our lecture topic in our ML eval class today. ✨ Evals have always played an important socio technical role in policy decision-making and surprisingly little changes when you add "AI" to the mix

Great webinar featuring Alessandro Acquisti on "The Economics of Privacy at a Crossroads." Alessandro has such a wide view of the privacy literature: both its history and inside & outside of economics. www.youtube.com/watch?v=ecc9...

TIL Minecraft has responsible AI lessons for kids education.minecraft.net//content/dam...

I was in Paris last week for the #AIActionSummit & was honoured to participate in the “AI in public interest” panel. 👇🏾 my thoughts & reflections on what AI in public interest is/isn’t & some concrete steps/initiatives for `bending the arc of AI towards the public interest' aial.ie/pages/aiparis/

When Toledo police rolled out their Fusus system, allowing officers to tap into the live feeds of privately owned cameras, they promised to only use the power in emergency situations. We obtained data that tells a very different story about when, and who, TPD watches. gizmodo.com/clearly-disc...

As someone who has reported on AI for 7 years and covered China tech as well, I think the biggest lesson to be drawn from DeepSeek is the huge cracks it illustrates with the current dominant paradigm of AI development. A long thread. 1/

How often does the FTC base its enforcement proceedings on third party research? Often! ftcreverse.engineering

oh, this is out and they make me sound like an angel but a really good read “We are not evaluating systems for some hypothetical, potential risks in the future,” Birhane says www.fastcompany.com/91238006/how...

Come work with @jessicahullman.bsky.social and I as a postdoc! We'll apply approaches from HCI, AI/ML, statistics, and decision theory to design new tools & methods for evaluating human-AI decision-making (e.g., tools to elicit, represent, & validate specifications of real-world decision problems).

tough morning for data brokers, great morning for people in addition to the CFPB FCRA rule (bsky.app/profile/benw...), the FTC announced a case against one who sold location data of visits to healthcare and places of worship, securing a ban on sale of this type of data www.ftc.gov/news-events/...

The proposed consent order with IntelliVision Technologies would prevent the company “from making misleading claims about its software. That would include any misleading statements about its performance identifying people of different genders, ethnicities, and skin tones.”

Something (someone) to be thankful for today — very excited to see this!!

Friends, this week marks a monumental step forward in my path towards pushing for a greater ecology of accountability in the age of AI. Thrilled to officially announce the AI Accountability Lab (AIAL) @theaial.bsky.social at Trinity College Dublin is launching today, Thursday Nov 28, 2024 aial.ie

Professors and researchers: are you booking speakers for the Spring semester? I have talks and workshops on data-driven investigations using crowdsourcing with WhatsApp, auditing black box AI systems and advanced web scraping. inspectelement.org/apis.html gist.github.com/yinleon/4e39...

Differential privacy is a SOTA approach for protecting individual privacy in data releases. But how it interacts w/common expectations & workflows can be hard to judge. New article for policy-makers w/P. Nanayakkara summarizes some gaps between theory & practice journals.sagepub.com/doi/abs/10.1...