Keeping chains-of-thought traces reflective of the models true reasoning would be very helpful for safety. Important work to explore the ways it may fail

New Google DeepMind paper: "Consistency Training Helps Stop Sycophancy and Jailbreaks" by @alexirpan.bsky.social, me, Mark Kurzeja, David Elson, and Rohin Shah. (thread)

[1/9] Excited to share our new paper "A Pragmatic View of AI Personhood" published today. We feel this topic is timely, and rapidly growing in importance as AI becomes agentic, as AI agents integrate further into the economy, and as more and more users encounter AI.

Evaluating the Infinite 🧵 My latest paper tries to solve a longstanding problem afflicting fields such as decision theory, economics, and ethics — the problem of infinities. Let me explain a bit about what causes the problem and how my solution avoids it. 1/N arxiv.org/abs/2509.19389

Do you have a PhD (or equivalent) or will have one in the coming months (i.e. 2-3 months away from graduating)? Do you want to help build open-ended agents that help humans do humans things better, rather than replace them? We're hiring 1-2 Research Scientists! Check the 🧵👇

digital-strategy.ec.europa.eu/en/policies/... The Code also has two other, separate Chapters (Copyright, Transparency). The Chapter I co-chaired (Safety & Security) is a compliance tool for the small number of frontier AI companies to whom the “Systemic Risk” obligations of the AI Act apply. 2/3

As models advance, a key AI safety concern is deceptive alignment / "scheming" – where AI might covertly pursue unintended goals. Our paper "Evaluating Frontier Models for Stealth and Situational Awareness" assesses whether current models can scheme. arxiv.org/abs/2505.01420

First position paper I ever wrote. "Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence" arxiv.org/abs/2506.23908 Background: I'd like LLMs to help me do math, but statistical learning seems inadequate to make this happen. What do you all think?

Can frontier models hide secret information and reasoning in their outputs? We find early signs of steganographic capabilities in current frontier models, including Claude, GPT, and Gemini. 🧵

Thought provoking

Are world models necessary to achieve human-level agents, or is there a model-free short-cut? Our new #ICML2025 paper tackles this question from first principles, and finds a surprising answer, agents _are_ world models… 🧵 arxiv.org/abs/2506.01622

Great to see serious work on non-agentic AI. I think it's an underappreciated direction: better for safety, society, and human meaning. LLMs show it's perfectly possible

When I realized how dangerous the current agency-driven AI trajectory could be for future generations, I knew I had to do all I could to make AI safer. I recently shared this personal experience, and outlined the scientific solution I envision @TEDTalks⤵️ www.ted.com/talks/yoshua...

Nice video about one of our recent papers, and some of its potential implications for AI agents www.youtube.com/watch?app=de...

Agency comes in degrees, and can vary along several dimensions: autonomy, efficacy, goal-complexity, and generality. Great paper helping us understand the different possibilities

METR task-time scaling critique: "the 4-minute mark for GPT-4 is completely arbitrary; you could probably put together one reasonable collection of word counting ... tasks with average human time of 30 seconds and another ... of 20 minutes where GPT-4 would hit 50% accuracy on each"

Generative AI tools used in art production should be evaluated by the broader art world of artists, art historians and curators, to integrate culturally-specific critique and to re-imagine these tools to suit the artists’ needs. dl.acm.org/doi/full/10....

interesting argument about manufacturing: "[with] good enough software and hardware to create [good] humanoid robots ..., we will also be able to create more task-specific hardware ... that can do those roles cheaper, faster, and better." perhaps the same is true for AI agents more generally?

I'm very impressed with the Sentinel newsletter: by far the best aggregator of global news I've found Expert forecasters filter for the events that actually matter (not just noise), and forecast how likely this is to affect eg war, pandemics, frontier AI etc Highly recommended!

Rival nations or companies sometimes choose to cooperate because some areas are protected zones of mutual interest—reducing shared risks without giving competitors an edge. Our paper in FAccT '25: How geopolitical rivals can cooperate on AI safety research. arxiv.org/abs/2504.12914

Excited that our paper "safety alignment should be made more than just a few tokens deep" was recognized as an #ICLR2025 Outstanding Paper! We identified a common root cause to many safety vulnerabilities and pointed out some paths forward to address it!

What if LLMs are sometimes capable of doing a task but don't try hard enough to do it? In a new paper, we use subtasks to assess capabilities. Perhaps surprisingly, LLMs often fail to fully employ their capabilities, i.e. they are not fully *goal-directed* 🧵 arxiv.org/abs/2504.118...

The 1st blackbox AI control paper uses a mixture of ML monitoring and editing to detect or block harm from malicious agents. However, blackbox control is one point in a space that varies with capability. Our new paper tracks how control might change along this trajectory. 🧵 arxiv.org/abs/2504.05259

Causality can help with AI safety in so many ways!

Super excited this giant paper outlining our technical approach to AGI safety and security is finally out! No time to read 145 pages? Check out the 10 page extended abstract at the beginning of the paper

𝐇𝐨𝐰 𝐜𝐚𝐧 𝐰𝐞 𝐩𝐞𝐫𝐟𝐞𝐜𝐭𝐥𝐲 𝐞𝐫𝐚𝐬𝐞 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬 𝐟𝐫𝐨𝐦 𝐋𝐋𝐌𝐬? Our method, Perfect Erasure Functions (PEF), erases concepts perfectly from LLM representations. We analytically derive PEF w/o parameter estimation. PEFs achieve pareto optimal erasure-utility tradeoff backed w/ theoretical guarantees. #AISTATS2025 🧵

Our work on concept discovery towards bridging the human-AI knowledge gap in AlphaZero has now been published in PNAS. As future AI systems become even more capable, we should be thinking of ways of utilizing them not only to perform tasks, but also to further our own knowledge and understanding.

🎉 Our Actionable Interpretability workshop has been accepted to #ICML2025! 🎉 > Follow @actinterp.bsky.social > Website actionable-interpretability.github.io @talhaklay.bsky.social @anja.re @mariusmosbach.bsky.social @sarah-nlp.bsky.social @iftenney.bsky.social Paper submission deadline: May 9th!

Google launched Gemini Pro Experimental 2.5, a “thinking model,” none of today’s generative AI models scored higher than 4% against a human average of 60% on the new ARC-AGI-2 test. And Jack Clark is becoming increasingly convinced that AGI is right around the corner. 🤔 www.cagii.org/blog/google-...

This metric from METR shows the length of tasks AI agents can complete has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. metr.org/blog/2025-03... 1/2

I have excellent news: our KU Leuven 𝗜𝗻𝘁𝗲𝗿𝗻𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗼𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗼𝗻 𝗟𝗮𝗿𝗴𝗲-𝗦𝗰𝗮𝗹𝗲 𝗔𝗜 𝗥𝗶𝘀𝗸𝘀 is now open for registration for the broader public! We received a lot of interesting talk submissions and we've made our selection of the best ones. It will take place on 𝟮𝟲-𝟮𝟴 𝗠𝗮𝘆 𝗶𝗻 𝗟𝗲𝘂𝘃𝗲𝗻, 𝗕𝗲𝗹𝗴𝗶𝘂𝗺.

We all want LLMs to collaborate with humans to help them achieve their goals. But LLMs are not trained to collaborate, they are trained to imitate. Can we teach LM agents to help humans by first making them help each other? arxiv.org/abs/2503.14481

This might be the most useful thing I have come across in social media - a personalized feed of academic papers filtered by your follower network! Highly recommend. #academicsky

Great work by Jack Foxabbott, Rohin Subramani, and my long-term collaborator @f-rhys-ward.bsky.social We always thought causality was useful for safe and ethical AI www.alignmentforum.org/s/pcdHisDEGL... This work extends casual models with subjective beliefs, incl. beliefs about others beliefs

My latest paper is just out on arXiv: : "Palatable Conceptions of Disembodied Being: Terra Incognita in the Space of Possible Minds". 🧵1/n

In real-life, agents with different subjective beliefs interact in a shared objective reality. They have higher-order beliefs about each other's beliefs and goals, which is required for phenomena involving theory-of-mind, like deception Our paper formalises this in causal models

Top notch. "discourse about large models as intelligent agents is fundamentally misconceived. ... Large Models should not be viewed primarily as intelligent agents, but as a new kind of cultural and social technology, allowing humans to take advantage of information other humans have accumulated."

👩‍💻 What is AGI...and should it be a goal? 😱 My podcast interview on this question is out! With @eryk.bsky.social and @borhane.bsky.social , interviewed by @justinhendrix.bsky.social for @techpolicypress.bsky.social : www.techpolicy.press/should-agi-r... Based on this paper: arxiv.org/abs/2502.03689

AISI has a new grant program for funding academic and nonprofit-affiliated research in (1) safeguards to mitigate misuse risk and (2) AI control and alignment to mitigate loss of control risk. Please apply! 🧵 www.aisi.gov.uk/grants

Next up in our ongoing AI Affect series: 👤 Tom Everitt (Google Deepmind) @tom4everitt.bsky.social 📢 "Agency as backwards causality" 🗓️ TODAY Tuesday, March 4, 3-4:30 PM 📍 Join us in person or online! DM for details.

We are excited to release a short course on AGI safety! The course offers a concise and accessible introduction to AI alignment problems and our technical / governance approaches, consisting of short recorded talks and exercises (75 minutes total). deepmindsafetyresearch.medium.com/1072adb7912c

Can we trust AI? “Everyone in the world is getting onto this brand new airplane that really has never been tested before. And it’s going to take off and it’s never going to land. It has to fly, forever.” – Stuart Russell, IASEAI President.

and you thought your arxiv paper had impact

Process based supervision done right, and with pretty CIDs to illustrate :)

Can AI boost human performance at training (even) better AI systems? This is an essential recursion to get right! The Rater Assist team at Google DeepMind describes their approach

🔦 @NeurIPS2024 spotlight paper we’re presenting today. Making AI systems more agentic is a hot research topic. But powerful agents bring worries about misalignment and loss of control. Can we measure how agentic an AI system is? 🧵

Excellent blue sky features summary

Personally, reviewing for NeurIPS a couple years back changed me as a reviewer. For one paper I rejected, I kept citing it throughout the year to people for a finding it had. This made me realise it was a good paper, it just had some easy targets for rejection.