[NeurIPS '25] Our oral slot and poster session on "Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research" are tomorrow, December 4! [https://arxiv.org/abs/2412.06966] Oral: 3:30-4pm PST, Upper Level Ballroom 20AB Poster 1307: 4:30:-7:30pm PST, Exhibit Hall C-E

I'm super excited about the 20th @wimlworkshop.bsky.social, which is taking place tomorrow in San Diego, co-located with @neuripsconf.bsky.social!!! 🎉 To celebrate, @jennwv.bsky.social and I recorded a podcast episode! Check it out here: www.microsoft.com/en-us/resear...

On my way to San Diego for #NeurIPS and the 20th anniversary of #WiML! To celebrate, @hannawallach.bsky.social and I recorded a podcast discussing WiML’s journey, our friendship and collaborations, research we're excited about, & advice we'd give our younger selves! www.microsoft.com/en-us/resear...

Alright, it's that time of year: Who all is going to @neuripsconf.bsky.social this year??? #NeurIPS2025 🤖☃️

Three exciting opportunities at @msftresearch.bsky.social in NYC!!! 🎉 Internship w/ FATE: apply.careers.microsoft.com/careers/job?... Internship w/ STAC on AI evaluation and measurement: apply.careers.microsoft.com/careers/job?... Postdoc w/ FATE: apply.careers.microsoft.com/careers/job?...

This is happening now!!!

1) (Tomorrow!) Wed 7/16, 11am-1:30 pm PT poster for "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" (E. Exhibition Hall A-B, E-503) Work led by @hannawallach.bsky.social + @azjacobs.bsky.social arxiv.org/abs/2502.00561

If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...

Generative language systems are everywhere, and many of them stereotype, demean, or erase particular social groups.

Alright, people, let's be honest: GenAI systems are everywhere, and figuring out whether they're any good is a total mess. Should we use them? Where? How? Do they need a total overhaul? (1/6)

I'm so excited this paper is finally online!!! 🎉 We had so much fun working on this with @emmharv.bsky.social!!! Thread below summarizing our contributions...

Exciting news: The Fairness, Accountability, Transparency and Ethics (FATE) group at Microsoft Research NYC is hiring a predoctoral fellow!!! 🎉 www.microsoft.com/en-us/resear...

Exciting news!!! This just got into @icmlconf.bsky.social as a position paper!!! 🎉 More updates to come as we work on the camera-ready version!!!

Reading - Evaluating Evaluations for GenAI from @hannawallach.bsky.social madesai.bsky.social afedercooper.bsky.social et al-This work dovetails with our work at @worldprivacyforum.bsky.social on measuring AI governance tools from governments, through privacy/ policy lens arxiv.org/pdf/2502.00561

At the #HEAL workshop, I'll present "Systematizing During Measurement Enables Broader Stakeholder Participation" on the ways we can further structure LLM evaluations and open them for deliberation. A project led by @hannawallach.bsky.social

2. Also Saturday, @amabalayn.bsky.social will represent our piece arguing that systematization during measurement enables broad stakeholder participation in AI evaluation. This came out of a huge group collaboration led by @hannawallach.bsky.social: bsky.app/profile/hann... heal-workshop.github.io

📣 New paper! The field of AI research is increasingly realising that benchmarks are very limited in what they can tell us about AI system performance and safety. We argue and lay out a roadmap toward a *science of AI evaluation*: arxiv.org/abs/2503.05336 🧵

⚫⚪ It's coming...SHADES. ⚪⚫ The first ever resource of multilingual, multicultural, and multigeographical stereotypes, built to support nuanced LLM evaluation and bias mitigation. We have been working on this around the world for almost **4 years** and I am thrilled to share it with you all soon.

Remember this @neuripsconf.bsky.social workshop paper? We spent the past month writing a newer, better, longer version!!! You can find it online here: arxiv.org/abs/2502.00561

🚨Postdoc Alert 🚨 The Computational Social Science group at Microsoft Research NYC (Jake Hofman, David Rothschild, Dan Goldstein) is hiring a postdoc! jobs.careers.microsoft.com/global/en/jo... Deadline: December 20, 2024

Microsoft's Computational Social Science group may have the opportunity to hire one researcher Senior: 0-3 yrs post PhD jobs.careers.microsoft.com/global/en/jo... Principal: 3+ yrs post PhD jobs.careers.microsoft.com/global/en/sh... Please note: our ability to hire this season is not certain

It's company holiday party season! Every year I start a thread of my favorite questions guaranteed to get you 20 minutes of lively conversation (as an introvert, this is how I thrive at parties). What are your favorites? Here are some of mine...

"there's a lot of qualitative work that goes into designing quantitative metrics" -- @azjacobs.bsky.social "how do we translate between benchmark performance and what it will really be like to use a model" -- Su Lin Blodgett

Super interesting panel discussion taking place right now at the Evaluating Evaluations workshop at @neuripsconf.bsky.social with amazing panelists @abeba.bsky.social, @azjacobs.bsky.social, Su Lin Blodgett, and Lee Wan Sie!!! #NeurIPS2024

And, as if that wasn't enough excitement for one day, we'll also be presenting a poster on "Red Teaming: Everything Everywhere All at Once" at the Safe GenAI @neuripsconf.bsky.social workshop from 3--5pm: neurips.cc/virtual/2024... #NeurIPS2024

Super excited for the Evaluating Evaluations workshop at @neuripsconf.bsky.social today!!! evaleval.github.io #NeurIPS2024 @msftresearch.bsky.social's FATE group, Sociotechnical Alignment Center, and friends will be presenting several papers there. See below for details...

Stop by the 345--430pm poster session at the Statistical Frontiers in LLMs @neuripsconf.bsky.social workshop today (Dec 14) to catch posters on the following papers... #NeurIPS2024

New paper on why machine "unlearning" is much harder than it seems is now up on arXiv: arxiv.org/abs/2412.06966 This was a huuuuuge cross-disciplinary effort led by @msftresearch.bsky.social FATE postdoc @grumpy-frog.bsky.social!!!

The AI Interdisciplinary Institute at the University of Maryland (AIM) is hiring 40 new faculty members in all areas of AI, particularly: - accessibility, - sustainability, - social justice, and - learning; building on computational, humanistic, or social scientific approaches to AI. >

I'll be giving a short talk on "Evaluating Generative AI Systems is a Social Science Measurement Challenge" (arxiv.org/abs/2411.10939) at 230pm today at the @msftresearch.bsky.social booth! #NeurIPS2024

@sosentos.bsky.social and I will be doing a fireside chat on "WiML and Me: Looking Back, Looking Forward" at the WiML Workshop today at 3:50pm in rooms 211--214! sites.google.com/wimlworkshop...

Reposting! Who is at @neuripsconf.bsky.social this week??? #NeurIPS2024

This year, the Affinity Events at NeurIPS are happening throughout the conference week and provide invaluable research contributions, mentorship opportunities, and community! We encourage everyone to actively participate. See schedule details in the thread below and here: tinyurl.com/fz3fesdy

One more of the @neuripsconf.bsky.social workshop papers from @msftresearch.bsky.social's FATE group, Sociotechnical Alignment Center, and friends is online! #NeurIPS2024

Super excited to announce that @msftresearch.bsky.social's FATE group, Sociotechnical Alignment Center, and friends have several workshop papers at next week's @neuripsconf.bsky.social. A short thread about (some of) these papers below... #NeurIPS2024

Alright, it's time for my annual tweet: Who all is going to @neuripsconf.bsky.social this year??? #NeurIPS2024 🤖☃️

Two exciting internship opportunities at @msftresearch.bsky.social in 2025!!! 🎉 With FATE in NYC: jobs.careers.microsoft.com/global/en/jo... With STAC (and collaborators in FATE) in NYC or Redmond focusing on AI evaluation and measurement: jobs.careers.microsoft.com/global/en/jo...