Delighted to see BigGen Bench paper receive the 🏆best paper award 🏆at NAACL 2025! BigGen Bench introduces fine-grained, scalable, & human-aligned evaluations: 📈 77 hard, diverse tasks 🛠️ 765 exs w/ ex-specific rubrics 📋 More human-aligned than previous rubrics 🌍 10 languages, by native speakers 1/

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

How should regulatory proposals adapt to the prevalence of general-purpose AI when the global geopolitical order is being reconfigured? @atoosakz.bsky.social, Deirdre K. Mulligan, @randomwalker.bsky.social, @alondra.bsky.social, & @shaynelongpre.bsky.social weigh in: youtu.be/cRsbjGFPJaM?...

🛬 in Singapore for #ICLR2025! DM me to catch up—but only if you have a local food/bar/event rec!

Thrilled our global data ecosystem audit was accepted to #ICLR2025! Empirically, it shows: 1️⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024). 2️⃣ YouTube is now 70%+ of speech/video data but could block third-party collection. 3️⃣ <0.2% of data from Africa/South America. 1/

📍EVENT: Day 2 of our “AI and Democracy” symposium will be kicking off shortly. Programming will begin with welcome remarks from George Deodatis @columbiaseas.bsky.social at 9:30am ET. #AIDemocraticFreedoms Watch the full event on our livestream here: www.youtube.com/watch?v=X1gj...

Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative! 🌍 18 languages (high-, mid-, low-) 📚 21k questions (55% require image understanding) 🧪 STEM, social science, reasoning, and practical skills

Panel 1: Regulating AI in a Time of Democratic Upheaval starts in approximately 5 minutes. Panelists: @atoosakz.bsky.social, @randomwalker.bsky.social, @alondra.bsky.social, and Deirdre K. Mulligan. Moderator: @shaynelongpre.bsky.social. #AIDemocraticFreedoms

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

Excited to speak at the workshop on Technical AI Governance in Vancouver this summer! #ICML2025

#AI is evolving fast, and so are its flaws. A fresh approach to finding and reporting AI bugs is long overdue. Great initiative by @shaynelongpre.bsky.social and team, transparency and accountability in AI development are essential! #AISafety #ResponsibleAI #AIEthics #MIT

After identifying major flaws in popular AI models, researchers are pushing for a new system to identify and report bugs.

Thank you @willknight.bsky.social for excellent coverage of our new proposal! www.wired.com/story/ai-res...

What are 3 concrete steps that can improve AI safety in 2025? 🤖⚠️ Our new paper, “In House Evaluation is Not Enough” has 3 calls-to-actions to empower evaluators: 1️⃣ Standardized AI flaw reports 2️⃣ AI flaw disclosure programs + safe harbors. 3️⃣ A coordination center for transferable AI flaws. 1/🧵

Very glad to join this paper organized by @shaynelongpre.bsky.social. Here's the paper itself: crfm.stanford.edu/2025/03/13/t...

Bringing transparency to the data used to train artificial intelligence mitsloan.mit.edu/ideas-made-t...

Thrilled to be at #AAAI2025 for our tutorial, “AI Data Transparency: The Past, Present, and Beyond.” We’re presenting the state of transparency, tooling, and policy, from the Foundation Model Transparency Index, Factsheets, the the EU AI Act to new frameworks like @MLCommons’ Croissant. 1/

Really excellent explainer by @shaynelongpre.bsky.social‬ that clearly lays out what's at stake in the "AI crawler wars"

Re: the FTC and the platforms We *should* be concerned about platform power over speech, but it isn’t censorship. As the Supreme Court said last year, the companies’ editorial decisions to moderate content are protected by the First Amendment. 1/

Let’s see if the algorithm and data remains transparent.

I compiled a list of resources for understanding AI copyright challenges (US-centric). 📚 ➡️ why is copyright an issue for AI? ➡️ what is fair use? ➡️ why are memorization and generation important? ➡️ how does it impact the AI data supply / web crawling? 🧵

Great point by @shaynelongpre.bsky.social on the AI crawler wars: "Unless we can nurture an ecosystem with different rules for different data uses, we may end up with strict borders across the web, exacting a price on openness and transparency." www.technologyreview.com/2025/02/11/1...

Our latest brief highlights the importance of independent evaluation for AI safety and accountability, addressing barriers and proposing safe harbors to protect third-party research. @klyman.bsky.social @shaynelongpre.bsky.social @sayash.bsky.social @peterhenderson.bsky.social

Appreciated this thoughtful op-ed about web crawlers and AI training… the biological metaphors are good as well as “invasive species”. by @shaynelongpre.bsky.social Link: www.technologyreview.com/2025/02/11/1...

"A future where only large companies can license or crawl critical web data would suppress competition and fail to serve real users or many of the copyright holders." Excellent @shaynelongpre.bsky.social www-technologyreview-com.cdn.ampproject.org/c/s/www.tech...

(not a lawyer) reading more into judge's opinion on westlaw ross case, some details that should be noted: 1. ross isn't Generative AI. it's a retrieval system where users type in a question & it returns relevant legal docs. there's no generated answer. see bsky.app/profile/dbam...

Great op-ed from @shaynelongpre.bsky.social on the effects AI — as a technology and as a market — is having on the web. www.technologyreview.com/2025/02/11/1...

“What I see here is media organizations that have the power to fight back against Trump but aren’t. ... This is a moment in which we need these organizations to hold the powerful to account, hold Trump to account,” @jameeljaffer.bsky.social @democracynow.org: www.democracynow.org/2025/2/11/ja...

I wrote a spicy piece on "AI crawler wars"🐞 in @technologyreview.com (my first op-ed)! While we’re busy watching copyright lawsuits & the EU AI Act, there’s a quieter battle over data access that affects websites, everyday users, and the open web. 🔗 www.technologyreview.com/2025/02/11/1... 1/

1/ Last week, we published the International AI Safety Report—supported by 30 nations plus the OECD, UN, and EU. Over 100 independent experts contributed. I’m thankful to play a small writing role, focusing on “Risks of Copyright.” 🔗 bit.ly/40Vm7Mu

Our updated Responsible Foundation Model Development Cheatsheet (250+ tools & resources) is now officially accepted to @tmlrorg.bsky.social (TMLR) 2025! It covers: - data sourcing, - documentation, - environmental impact, - risk eval - model release & licensing - ++

Dear journalists: Now, more than ever, ✨The lawyers need to see the actual court filings.✨ Yes, I know you summarized. Yes, I know your non-attorney readers don't care. But we need to see the documents. The actual documents. If you're pulling docs from PACER, use RECAP & give us a link. Please.

It feels tactical to say this but if you like a text someone has written here consider sharing it. Likes on Bsky signal your approval, but they don’t bump it or amplify it because there’s no algorithm. *Sharing* independent writer’s work helps us find and grow an audience!

🪶 Some thoughts on DeepSeek, OpenAI, and the copyright battles: This isn’t the first time OpenAI has accused a Chinese company of breaking its Terms and training on ChatGPT outputs. Dec 2023: They suspended ByteDance’s accounts. 1/

Will AI eat itself? I appeared in BBC Radio 4’s The Artificial Human, alongside NYU Prof. Julia Kempe, to discuss: • The explosion of AI-generated content • Synthetic data → model collapse or helpful? • The data economy & the vital role of humans www.bbc.co.uk/sounds/play/...

apply here to join the accountability lab (@aial.ie)! aial.ie/pages/hiring/ currently accepting applications for phd and postdoc positions

I wrote up a quick reaction (at @justsecurity.org) to Trump’s executive order on free speech. www.justsecurity.org/106608/free-...

Trump’s #TikTok Executive Order aims to consolidate his power over the digital public sphere. Our statement from @ramyakrishnan.bsky.social below. knightcolumbia.org/content/trum...

Excited our new report will appear at #AAAI 2025 The #DEFCON2024 @aivillage.bsky.social Generative Red Team 2 (GRT2) Case Study spanned: ⚔️495 hackers, against AI2’s Olmo + WildGuard 🐞200 model flaw reports 💰$7k+ paid bounties Check it out! 🔗 arxiv.org/pdf/2410.12104

Just shared my thoughts with @technologyreview.com on AI's data problem. With most training data coming from English-language sources, we're building AI systems that perpetuate a Western-centric worldview. Data diversity isn't a checkbox: it determines how AI systems represent daily life globally.

Data is not a "naturally occurring resource", but "something that is created through particular processes” S. Myers West. These processes are dominated by Big Tech -> AI models are very asymmetrical in terms of access & representation. @melissahei.bsky.social www.technologyreview.com/2024/12/18/1...

kicking off 2025 with our OLMo 2 tech report while payin homage to the sequelest of sequels 🫡 🚗 2 OLMo 2 Furious 🔥 is everythin we learned since OLMo 1, with deep dives into: 🚖 stable pretrain recipe 🚔 lr anneal 🤝 data curricula 🤝 soups 🚘 tulu post-train recipe 🚜 compute infra setup 👇🧵

Looking for people to follow who will be thinking through some of the toughest free speech and new technology questions in the coming year? Check out the Knight Institute’s starter pack of scholars, litigators, and tech policy experts. go.bsky.app/5D94PgW

✨ Excited to share our latest work from The Data Provenance Initiative ☸️ This is the most comprehensive audit of multimodal training data, auditing ~4000 datasets between 1990 and 2024, and covering more than 400 unique tasks in 608 languages! 🧵 1/n

Thank you @melissahei.bsky.social for covering the Data Provenance Initiative's new study on 🌟 Multimodal data 🌟. Our study finds: - A concentration of data power - Western-centricity isn't improving for 10+ years Read more! www.dataprovenance.org/Multimodal_D...

Hehe been shouting this from the rooftops for a while now 🙃 Concentration of data deals and unaccountable uses of interaction and uploaded data by AI providers (incl. through their other services) are as much a threat to AI governance as compute concentration, and they're a lot more insidious...

Data isn’t a naturally occurring resource; it’s shaped by the intentions and design of those collecting it - this means that tech firms’ advantage in data collection reshapes infrastructures in their own interests. Read more in this really critical piece by @melissahei.bsky.social:

New research reveals a worrying trend: AI's data practices risk concentrating power overwhelmingly in the hands of dominant technology companies. With analysis from @shaynelongpre.bsky.social @sarahooker.bsky.social @smw.bsky.social @giadapistilli.com www.technologyreview.com/2024/12/18/1...

Touching down in Vancouver 🛬 for #NeurIPS2024! I'll be presenting our "Consent in Crisis" work on the 11th: arxiv.org/abs/2407.14933 Reach out to catch up or chat about: - Training data / methods - AI uses & impacts - Multilingual scaling

Is MMLU Western-centric? 🤔 As part of a massive cross-institutional collaboration: 🗽Find MMLU is heavily overfit to western culture 🔍 Professional annotation of cultural sensitivity data 🌍 Release improved Global-MMLU 42 languages 📜 Paper: arxiv.org/pdf/2412.03304 📂 Data: hf.co/datasets/Coh...