With, e.g., OpenAI planning over 1T in commitments in the next few years, it increasingly seems that one of two bad things will inevitably happen: a bubble bursting or the concentration of obscene levels of power in tech. I don't see how this ends well. techcrunch.com/2025/11/06/...

Taking AI safety seriously means taking open-weight model safety seriously. Unfortunately, the AI safety field has historically mostly worked with closed models in mind. Here, I explain how we can meet new challenges from open models. www.youtube.com/watch?v=VWk3...

🧵🧵🧵 In the past few months, I have looked at hundreds, maybe thousands, of AI porn images/videos (for science). Here's what I learned from our investigation of over 50 platforms, sites, apps, Discords, etc., while writing this paper. papers.ssrn.com/sol3/papers...

🧵 I think people often assume that AI images/video will get harder to distinguish from natural ones over time with better models. In most (non-adversarial) cases, I expect the opposite will often apply...

Excited that our paper has been on SSRN for 8 days, but became SSRN's most downloaded paper of the past 60 days in two ejournal categories. Glad about this -- I think this is one of the more important projects I've worked on. papers.ssrn.com/sol3/papers....

UK AISI is hiring for a technical research role on open-weight model safeguards. www.aisi.gov.uk/careers

Did you know that one base model is responsible for 94% of model-tagged NSFW AI videos on CivitAI? This new paper studies how a small number of models power the non-consensual AI video deepfake ecosystem and why their developers could have predicted and mitigated this.

Here are my current favorite ideas for how to improve tamper-resistant ignorance/unlearning in LLMs. Shamelessly copied from a slack message.

🌵🐎🤠🏜️🐄 Here's a roundup of some key papers on data filtering & safety. Tl;DR -- Filtering harmful training data seems to effectively make models resist attacks (incl. adv. fine-tuning), but only when the filtered content is 'hard to learn' from the non-filtered content 🧵

I’m pleased to share the Second Key Update to the International AI Safety Report, which outlines how AI developers, researchers, and policymakers are approaching technical risk management for general-purpose AI systems. (1/6)

The leaked executive order has me wondering if the term "regulatory capture" has any meaning anymore. It appears that state AI bills -- many of which big tech has fought tooth and nail to prevent -- are categorically regulatory capture.

Based on what I've seen lately, it sounds like rebuttals for @iclr_conf are a mess. But in case it makes your life easier, feel free to copy or adapt my rebuttal template linked here. docs.google.com/document/d/1...

🚨New paper🚨 From a technical perspective, safeguarding open-weight model safety is AI safety in hard mode. But there's still a lot of progress to be made. Our new paper covers 16 open problems. 🧵🧵🧵

🚨New paper🚨 From a technical perspective, safeguarding open-weight model safety is AI safety in hard mode. But there's still a lot of progress to be made. Our new paper covers 16 open problems. 🧵🧵🧵

I've essentially stopped paying attention to companies' AI eval reports. They're way too easy to game and, at this point, probably lack meaningful construct validity. I'm increasingly persuaded that the only quantitative measures that matter anymore are usage stats & profit.

This summer, OpenAI, Anthropic, and GDM warned that their new models were nearing key risk thresholds for novice uplift on dangerous tasks. Now that Moonshot claims Kimi K2 Thinking is SOTA, it seems, uh, less than ideal that it came with zero reporting related to safety/risk.

Most frontier Western/US AI models are proprietary. But most frontier Eastern/Chinese models have openly available weights. Why? Is it because more Chinese companies are "fast followers" who find their niche by making open models? Is it cultural? Do Eastern/Chinese cultures value open tech more?

How might the world look after the development of AGI, and what should we do about it now? Help us think about this at our workshop on Post-AGI Economics, Culture and Governance! We’ll host speakers from political theory, economics, mechanism design, history, and hierarchical agency. post-agi.org

Our proposal for new AI watermarking characters is officially in the Unicode document register for proposed additions. 🤞 unicode.org/L2/L2025/252... t.co/yJfp8ezU64

🧵🧵🧵 Do you ever hear people saying that it's important to assess AI systems based on their "marginal risk"? Of course -- that's obvious. Nobody would ever dispute that. So then why are we saying that? Maybe it's a little too obvious...

Technologies like synthetic data, evaluations, and red-teaming are often framed as enhancing AI privacy and safety. But what if their effects lie elsewhere? In a new paper with @realbrianjudge.bsky.social at #EAAMO25, we pull back the curtain on AI safety's toolkit. (1/n) arxiv.org/pdf/2509.22872

🧵🧵🧵 Do you ever hear people saying that it's important to assess AI systems based on their "marginal risk"? Of course -- that's obvious. Nobody would ever dispute that. So then why are we saying that? Maybe it's a little too obvious...

In our Nature article, @yaringal.bsky.social and I outline how building the technical toolkit for open-weight AI model safety will be key to both accessing the benefits and mitigating the risks of powerful open models. www.nature.com/articles/d41...

Don't forget that in AI, "sycophancy," "pandering," "personalized alignment," "steerable alignment," and "user alignment" all describe exactly the same thing.

Almost 2 years out from my paper with Carson Ezell et al. titled "Black-Box Access is Insufficient for Rigorous AI Audits," it's cool to see that AI companies are starting to report on [internal] evals that use fine-tuning or interp-based methods.

A monumental milestone for AI governance: www.gov.ca.gov/2025/09/29/g...

LLM agents are optimized for thumbs-up instant gratification. RLHF -> sycophancy We propose human agency as a new alignment target in HumanAgencyBench, made possible by AI simulation/evals. We find e.g., Claude most supports agency but also most tries to steer user values 👇 arxiv.org/abs/2509.08494

I'll be leading a MATS stream this winter with a focus on technical AI governance. You can apply here by October 2! www.matsprogram.org/apply

📌📌📌 I'm excited to be on the faculty job market this fall. I just updated my website with my CV. stephencasper.com

Here is a riddle I came up with for a draft to illustrate the differences between normal chat models and reasoning models. Can you figure it out? Dark as night in the morning light. I live high until I am ground. I sit dry until I am drowned. What am I?

Research on AI "sandbagging" is getting more popular recently. In this 🧵, I'll give some reasons that I think it's not a useful research paradigm. TL;DR, I think it's a confusing reframing of fairly well studied and previously solved problems.

There have been a couple cool pieces up recently debunking the "China is racing on AI, so the US must too" narrative. time.com/7308857/chin... papers.ssrn.com/sol3/papers....

A personal update: - I just finished my 6-month residency at UK AISI. - I'm going back to MIT for the final year of my PhD. - I'm on the postdoc and faculty job markets this fall!

Some good thoughts on our paper from Jack Clark's ImportAI newsletter. I'll share a couple of thoughts on this here 🧵🧵 t.co/nHMFKXF4B8

Here are a couple of slides that I presented yesterday at #aitechgov about open-weight model risk management.

"Our study therefore shows that data filtration can be a powerful tool in helping developers balance safety and innovation in open-source AI." www.ox.ac.uk/news/2025-08...

"The results suggest policymakers may need to question one of the AI industry’s long-established narratives." www.washingtonpost.com/newsletter/p...

Are you afraid of LLMs teaching people how to build bioweapons? Have you tried just... not teaching LLMs about bioweapons? @eleutherai.bsky.social‬ and the UK AISI joined forces to see what would happen, pretraining three 6.9B models for 500B tokens and producing 15 total models to study

🧵 New paper from UK AISI x @eleutherai.bsky.social rai.bsky.social‬ that I led with @kyletokens.bsky.social y.social���: Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.

Haha, yup. arxiv.org/abs/2508.05338

🚨 New report 🚨 What does the public think about **specific** AI policy proposals? We asked 300 working-class adults in CA, IL, and NY. zenodo.org/records/1656...

I just finished Empire of AI. Usually, the story of AI is told as a story about progress in capabilities, driven by R&D. But in this book, @_KarenHao tells it as a story about power, driven by people pursuing it.

Hi I am in Vancouver for ICML and THE technical AI governance workshop. Message me if you’d like to talk about technical governance research or UK AISI’s safeguards work.

The Singapore Consensus is on arXiv now -- arxiv.org/abs/2506.20702 It offers: 1. An overview of consensus technical AI safety priorities 2. An example of widespread international collab & agreement

Making LLMs robust to tampering attacks might be one of the biggest priorities for safeguards research. @scasper.bsky.social argues this resistance may predict & upper-bound overall AI robustness, making it a key safety priority over the next year.

Let me know if you’d like to talk at @facct.bsky.social!

It's hard to plan for AGI without knowing what outcomes are even possible, let alone good. So we’re hosting a workshop! Post-AGI Civilizational Equilibria: Are there any good ones? Vancouver, July 14th www.post-agi.org Featuring: Joe Carlsmith, @richardngo.bsky.social‬, Emmett Shear ... 🧵

I recently learned that AI-generated text is sometimes sent to users with Unicode watermarks using nonstandard Unicode characters. But it also would seem nice to have an invisible Unicode character that means "Author does not consent to AI training on this document."

YGTBFKM

🎥 Singapore Alignment Workshop videos are live! Hear from @yoshuabengio.bsky.social @scasper.bsky.social @zicokolter.bsky.social @sivareddyg.bsky.social @shaynelongpre.bsky.social @markbrakel.bsky.social @achan96.bsky.social @teganmaharaj.bsky.social + more. Full playlist 👇