I've essentially stopped paying attention to companies' AI eval reports. They're way too easy to game and, at this point, probably lack meaningful construct validity. I'm increasingly persuaded that the only quantitative measures that matter anymore are usage stats & profit.

This summer, OpenAI, Anthropic, and GDM warned that their new models were nearing key risk thresholds for novice uplift on dangerous tasks. Now that Moonshot claims Kimi K2 Thinking is SOTA, it seems, uh, less than ideal that it came with zero reporting related to safety/risk.

Most frontier Western/US AI models are proprietary. But most frontier Eastern/Chinese models have openly available weights. Why? Is it because more Chinese companies are "fast followers" who find their niche by making open models? Is it cultural? Do Eastern/Chinese cultures value open tech more?

How might the world look after the development of AGI, and what should we do about it now? Help us think about this at our workshop on Post-AGI Economics, Culture and Governance! We’ll host speakers from political theory, economics, mechanism design, history, and hierarchical agency. post-agi.org

Our proposal for new AI watermarking characters is officially in the Unicode document register for proposed additions. 🤞 unicode.org/L2/L2025/252... t.co/yJfp8ezU64

🧵🧵🧵 Do you ever hear people saying that it's important to assess AI systems based on their "marginal risk"? Of course -- that's obvious. Nobody would ever dispute that. So then why are we saying that? Maybe it's a little too obvious...

Technologies like synthetic data, evaluations, and red-teaming are often framed as enhancing AI privacy and safety. But what if their effects lie elsewhere? In a new paper with @realbrianjudge.bsky.social at #EAAMO25, we pull back the curtain on AI safety's toolkit. (1/n) arxiv.org/pdf/2509.22872

🧵🧵🧵 Do you ever hear people saying that it's important to assess AI systems based on their "marginal risk"? Of course -- that's obvious. Nobody would ever dispute that. So then why are we saying that? Maybe it's a little too obvious...

In our Nature article, @yaringal.bsky.social and I outline how building the technical toolkit for open-weight AI model safety will be key to both accessing the benefits and mitigating the risks of powerful open models. www.nature.com/articles/d41...

Don't forget that in AI, "sycophancy," "pandering," "personalized alignment," "steerable alignment," and "user alignment" all describe exactly the same thing.

Almost 2 years out from my paper with Carson Ezell et al. titled "Black-Box Access is Insufficient for Rigorous AI Audits," it's cool to see that AI companies are starting to report on [internal] evals that use fine-tuning or interp-based methods.

A monumental milestone for AI governance: www.gov.ca.gov/2025/09/29/g...

LLM agents are optimized for thumbs-up instant gratification. RLHF -> sycophancy We propose human agency as a new alignment target in HumanAgencyBench, made possible by AI simulation/evals. We find e.g., Claude most supports agency but also most tries to steer user values 👇 arxiv.org/abs/2509.08494

I'll be leading a MATS stream this winter with a focus on technical AI governance. You can apply here by October 2! www.matsprogram.org/apply

📌📌📌 I'm excited to be on the faculty job market this fall. I just updated my website with my CV. stephencasper.com

Here is a riddle I came up with for a draft to illustrate the differences between normal chat models and reasoning models. Can you figure it out? Dark as night in the morning light. I live high until I am ground. I sit dry until I am drowned. What am I?

Research on AI "sandbagging" is getting more popular recently. In this 🧵, I'll give some reasons that I think it's not a useful research paradigm. TL;DR, I think it's a confusing reframing of fairly well studied and previously solved problems.

There have been a couple cool pieces up recently debunking the "China is racing on AI, so the US must too" narrative. time.com/7308857/chin... papers.ssrn.com/sol3/papers....

A personal update: - I just finished my 6-month residency at UK AISI. - I'm going back to MIT for the final year of my PhD. - I'm on the postdoc and faculty job markets this fall!

Some good thoughts on our paper from Jack Clark's ImportAI newsletter. I'll share a couple of thoughts on this here 🧵🧵 t.co/nHMFKXF4B8

Here are a couple of slides that I presented yesterday at #aitechgov about open-weight model risk management.

"Our study therefore shows that data filtration can be a powerful tool in helping developers balance safety and innovation in open-source AI." www.ox.ac.uk/news/2025-08...

"The results suggest policymakers may need to question one of the AI industry’s long-established narratives." www.washingtonpost.com/newsletter/p...

Are you afraid of LLMs teaching people how to build bioweapons? Have you tried just... not teaching LLMs about bioweapons? @eleutherai.bsky.social‬ and the UK AISI joined forces to see what would happen, pretraining three 6.9B models for 500B tokens and producing 15 total models to study

🧵 New paper from UK AISI x @eleutherai.bsky.social rai.bsky.social‬ that I led with @kyletokens.bsky.social y.social���: Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.

Haha, yup. arxiv.org/abs/2508.05338

🚨 New report 🚨 What does the public think about **specific** AI policy proposals? We asked 300 working-class adults in CA, IL, and NY. zenodo.org/records/1656...

I just finished Empire of AI. Usually, the story of AI is told as a story about progress in capabilities, driven by R&D. But in this book, @_KarenHao tells it as a story about power, driven by people pursuing it.

Hi I am in Vancouver for ICML and THE technical AI governance workshop. Message me if you’d like to talk about technical governance research or UK AISI’s safeguards work.

The Singapore Consensus is on arXiv now -- arxiv.org/abs/2506.20702 It offers: 1. An overview of consensus technical AI safety priorities 2. An example of widespread international collab & agreement

Making LLMs robust to tampering attacks might be one of the biggest priorities for safeguards research. @scasper.bsky.social argues this resistance may predict & upper-bound overall AI robustness, making it a key safety priority over the next year.

Let me know if you’d like to talk at @facct.bsky.social!

It's hard to plan for AGI without knowing what outcomes are even possible, let alone good. So we’re hosting a workshop! Post-AGI Civilizational Equilibria: Are there any good ones? Vancouver, July 14th www.post-agi.org Featuring: Joe Carlsmith, @richardngo.bsky.social‬, Emmett Shear ... 🧵

I recently learned that AI-generated text is sometimes sent to users with Unicode watermarks using nonstandard Unicode characters. But it also would seem nice to have an invisible Unicode character that means "Author does not consent to AI training on this document."

YGTBFKM

🎥 Singapore Alignment Workshop videos are live! Hear from @yoshuabengio.bsky.social @scasper.bsky.social @zicokolter.bsky.social @sivareddyg.bsky.social @shaynelongpre.bsky.social @markbrakel.bsky.social @achan96.bsky.social @teganmaharaj.bsky.social + more. Full playlist 👇

There has been a recent rise of open models specifically fine-tuned to lack safeguards. I think it's interesting and politically revealing that the fine-tuners of these models call them "uncensored" instead of alternatives like "unrestricted" or "helpful-only."

🧵A challenge with H.R.1's moratorium on state-level AI regulation is that it's not clear what "regulating" AI means. The bill doesn't define "regulation" in this context.

🧵 🧵 🧵 I think that making LLMs resistant to exhibiting harmful behavior under few-shot tampering attacks might be one of the most useful goals for AI risk management in the next year.

Hi, I’m at ICLR — let’s talk about AI evals, safeguards, and technical governance.

🚨New paper: Current reports on AI audits/evals often omit crucial details, and there are huge disparities between the thoroughness of different reports. Even technically rigorous evals can offer little useful insight if reported selectively or obscurely. Audit cards can help.

Stoked that this work with Ariba Khan and @dhadfieldmenell.bsky.social was accepted to @facct.bsky.social See the preprint for more about how cultural alignment evals can be misleading. Cultural alignment is not a model property -- but a model + eval-context property arxiv.org/abs/2503.08688

New Pew Research AI expert survey is interesting: pewresearch.org/internet/202...

I had ChatGPT make this cartoon illustrating your three career paths if you work on AI risk management.

📣We’re thrilled to announce the first workshop on Technical AI Governance (TAIG) at #ICML2025 this July in Vancouver! Join us (& this stellar list of speakers) in bringing together technical & policy experts to shape the future of AI governance! www.taig-icml.com

I recommend these two tools to quickly browse CS papers every day. - This link to see daily CS papers: vjunetxuuftofi.github.io/arxivredirect/ - This plugin to auto-highlight keywords: chromewebstore.google.com/detail/highl...

As AI becomes increasingly transformative, we need to rethink how we approach safety – not as a technical alignment problem, but as an ongoing, unsexy struggle.  stephencasper.com/reframing-ai...

This report will likely be a key force in California legislation this year. It discusses a lot including evidence-seeking policy, transparency, third-party assessments, and liability. Public comment is open until April 8. cafrontieraigov.org

I wish Google Scholar allowed researchers to pin 1-3 papers of their choosing to appear at the top of their Google Scholar page.

AI agents are currently unreliable and still not that useful for doing long-horizon tasks all by themselves. Meanwhile, the systems that seem arguably the most popular/impactful are assistants like Cursor. Seems like a good sign for the assistance paradigm. openreview.net/forum?id=DFI...