With SOTA defenses LLMs can be difficult even for experts to exploit. Yet developers often compromise on defenses to retain performance (e.g. low-latency). This paper shows how these compromises can be used to break models – and how to securely implement defenses.

So many great talks from the Singapore Alignment Workshop -- I look forward to catching up on those that I missed in person!

As I say in the video, innovation vs safety is a false dichotomy -- do check out great ideas from our speakers for how innovation can enable effective policy in the video 👇 and initial talk recordings!

AI control is one of the most exciting new research directions; excited to have the videos from ControlConf, the world's first control-specific conference. Tons of great material both intros & diving into specific areas!

AI security needs more than just testing, it needs guarantees. Evan Miyazono calls for broader adoption of formal proofs, suggesting a new paradigm where AI produces code to meet human specifications.

Had a great time at the Singapore Alignment Workshop earlier this week -- fantastic start to the ICLR week! My only complaint is I missed many of the excellent talks because I was having so many interesting conversations. Looking forward to the videos to catch up!

ControlConf 2025 Day 2 delivered! From robust evals to security R&D & moral patienthood, we covered the edge of AI control theory and practice. Thanks to Ryan Greenblatt, Rohin Shah, Alex Mallen, Stephen McAleer, Tomek Korbak, Steve Kelly & others for their insights.

My biggest complaint with the AI Security Forum was too much great content across the three tracks. Looking forward to catching up on the talks I missed with the videos 👇

Excited to meet others working on or interested in alignment at the Alignment Workshop Open Social before ICLR!

Excited to see people before ICLR at Alignment Workshop Singapore!

Humans sometimes cheat at exams -- might AIs do the same? Unique challenge to evaluating intelligent systems.

AI agents can start VMs, buy things, send e-mails, etc. AI control is a promising way to prevent harmful agent actions -- whether by accident, due to adversarial attack, or the systems themselves trying to subvert controls. Apply to the world's first control conference 👇

Since joining FAR.AI in June, Lindsay has delivered amazing events like the Alignment Workshop Bay Area and Paris Security Forum. Welcome to the team!

Evaluations are key to understanding AI capabilities and risks -- but which ones matter? Enjoyed Soroush's talk exploring these issues!

Excited to have Annie join our team, and help produce a 200-person event in her first month! We're growing across operations and technical roles -- check out opportunities 👇

I had great conversations at the AI Security Forum -- it's exciting to see people from cybersec, hardware root of trust, and AI come together to come up with creative solutions to boost AI security.

Formal verification has a lot of exciting applications, especially in the age of LLMs: e.g. can LLMs output programs with proofs of correctness? However formally verifying neural network behavior in general seems intractable -- enjoyed Zac's talk on limitations.

Many eyes on code secures critical open-source code -- we similarly need independent scrutiny of AI models to catch issues and have trust in the models. A safe harbor for evaluation as proposed by @shaynelongpre.bsky.social could enable an independent testing ecosystem.

Deceptive alignment is one of the more pernicious safety risks. I used to find it far fetched but LLMs are very good at persuasion so have the capability -- it's just a question of whether it's incentivized during training. Great to see work towards detecting deception!

Gradient routing can influence where a neural network learns a particular skill -- useful for interpretability and control.

Open-weight models are more flexible, decentralize power in AI and power research. But their flexibility also allows them to be misused: e.g. the surge in gen AI phishing. Tamper-resistant safeguards could give best of both worlds! Challenging but very important research area.

"Balancing helpfulness and safety is a real challenge for AI agents in mobile environments." Kimin Lee from KAIST introduces MobileSafetyBench—a tool for testing AI safety in mobile devices.

🛡️🔐 Paris AI Security Forum 2025 on Feb 9! 🇫🇷 Join 150+ engineers, researchers & policymakers from leading labs, academia & government to discuss challenges in AI system security & safety. Hear from speakers at RAND, Palo Alto Networks, Dreadnode, Pattern Labs & more. 🔗👇

“We found that if you ask the LLM, surprisingly it always says that I'm 100% confident about my reasoning.” Chirag Agarwal examines the (un)reliability of chain-of-thought reasoning, highlighting issues in faithfulness, uncertainty & hallucination.

Looking forward to the AI Security forum on Feb 9th -- just before Paris AI Action Summit and two days after IASEAI. AI opens up new classes of security vulnerabilities and AI model weights may be one of the most valuable assets making AI infosec timely & fascinating!

Empirical testbeds for ways in which AI systems can misbehave are crucial to make technical progress; great to hear from Evan on his work on model organisms.

Scalable oversight is key to aligning models on tasks where AIs are better than most (or all) evaluators -- excited to see @_julianmichael_'s empirical work on this!

If we want AI policy informed by science, we must build rigorous, third-party model evaluation tools. Stephen Casper advocates for model manipulation attacks to reveal hidden AI risks, especially for open-weight and closed-source models.

"Are models capable of serious harm still stupid enough to fall prey to simple jailbreaks?" Adam Gleave examines if scaling alone can solve AI vulnerabilities. While adversarial training boosts resilience, it’s still orders of magnitude less efficient than attacks.

China classifies AI safety as a national security issue with cybersecurity, biological security & natural disasters. Kwan Yee Ng outlined China’s policies: model registration, safety checks for gen AI, and AGI safety pilots in Beijing, Shanghai, etc. #AlignmentWorkshop

Reminder: Come visit us at #NeurIPS2024 11am-2pm today! 🔬 InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques 🎓 Rohan Gupta · Iván Arcuschin Moreno · Thomas Kwa · Adrià Garriga-Alonso

I often get asked how to stay in the loop on what FAR.AI is working on. We now have a a newsletter! 📷 This will be low volume (once a quarter) summarizing our key achievements and a sneak peak of future plans. Sign up 👇 to keep updated!

Experiments are critical for rigorous interpretability methods making causally valid predictions: enjoyed Atticus's talk on limitations of current methods and ways to improve them.

Excited to have four papers at NeurIPS across reward overoptimization, generalization, & inteerpretability. Won't be making it to NeurIPS myself this year but do chat to my colleagues Adrià Garriga-Alonso and Aaron Tucker if you're attending!

Optimized misalignment is one of the AI threat models with the greatest potential risk but has seen relatively little work investigating it -- loved this overview by Anca on this topic.

Knowing what AI systems can and can't do is crucial to making informed development, deployment and governance decisions. Really enjoyed Beth of @metr.org's summary of their recent evaluations work!