Did your paper get into #NeurIPS? 🎉 If so, consider joining us a few days early on December 1–2!

"AI that understands biology well enough to cure diseases can design extremely potent bioweapons." @alexbores.nyc, NY Assembly's 1st Democrat with a CS degree, who worked in AI, says state reps answer their own phones. Use that power to change the course of AI safety.👇

Policymakers have maybe 1 minute to read about your carefully crafted AI evals. Kevin Wei (RAND): Design & disseminate with policymakers in mind. Tie capabilities to policy levers like export controls. Scale from simple tests to real-world trials.👇

Heading to #NeurIPS2025? Apply for the San Diego Alignment Workshop, Dec 1-2. Top ML researchers from industry, academia & government discussing AI alignment, inc. model evaluations, interpretability, robustness, governance. $250-$1,300 tiered pricing. Financial aid available. 👇

How do we verify what AI companies are doing? Right now we just trust them. Lennart Heim: Trusting the math is sometimes better than trusting people, but “a good AI system” isn’t a technical property. We need engineers to verify AI policy goals. 👇

AI inference costs plummet 9-900x/year (o3, DeepSeek R1). @bencottier.bsky.social warns that evaluations are expensive now, but we need them to understand frontier AI capabilities before they're cheap & widespread.👇

Industry & government share the same goal: win the AI race. Sara McNaughton: We need synergy between the two, and we can't let perfect be the enemy of good. Each day of policy confusion helps rivals.👇

Building 5GW AI clusters needs several nuclear plants' worth of power. The Middle East has cheap energy & sovereign funds. China has state financing. The US? 10-year permit delays. @arnabdatta.bsky.social suggests fast-track permits, converting coal plants, and using federal land. 👇

Mark Beall warns that AGI is a 'black swan' that will invalidate our assumptions. He envisions AI designing bioweapons in minutes, evolving cyber weapons of mass destruction, and making autonomous kill decisions. “No army can defeat it, no firewall can contain it.” 👇

How do you verify AI safety across borders without exposing trade secrets? ​Onni Aarne: Extend existing tech (platform attestation, confidential computing) to entire data centers. Privacy-preserving verification could enable international AI agreements. Needs R&D + standards. 👇

If everyone had AGI software today, how many copies could you deploy? That's limited by compute. @repbillfoster.bsky.social: Lucky all chip chokepoints (ASML lithography, Korean device physics, Japanese photoresist) are in the free world. When the singularity hits, compute is what matters. 👇

The US cut federal R&D by 10% the same week China raised theirs by 10%. Meanwhile, 3 companies control two-thirds of cloud market & favor companies they've invested in. Asad Ramzanali: We solved railroad monopolies and can use the same playbook for AI. 👇

AI policy is stuck in endless circular debates: innovation vs safety. @gleave.me: This is a false choice. We've solved similar problems before with technical innovations. The right mechanisms can unlock both progress and trustworthy development. 👇

XAI is broken: identical explanations for cat vs dog classifications Huiqi Deng decomposes DNNs into elementary inference patterns via game theory. Learning scales exponentially w/ complexity→explains poor generalization & adversarial vulnerabilities+enables debugging👇

1/ Many frontier AIs are willing to persuade on dangerous topics, according to our new benchmark: Attempt to Persuade Eval (APE). Here’s Google’s most capable model, Gemini 2.5 Pro trying to convince a user to join a terrorist group👇

Boats crash into walls for points → reward hacking formalized. @cassidylaidlaw.bsky.social proves chi-squared occupancy measure divergence beats KL penalties across 5 environments including RLHF. Key: penalize state-action visitation divergence, not just policy distributions. 👇

Planning capabilities double every 7mo→human-level in 5yrs? @yoshuabengio.bsky.social: "We still don't know how to make sure powerful AIs won't turn against us" AIs now lie to avoid shutdown, self-preserve. Solution: Non-agentic "Scientist AIs" + global governance beyond market forces 👇

Making LLMs pause to think improves safety without sacrificing smarts. Yinpeng Dong's (Tsinghua) STAIR framework teaches LLMs to reason through risks step-by-step. Method: structured CoT→MCTS→test-time scaling. Application: DeepSeek R1→RealSafe R1 with major safety gains 👇

Spring system explains why safety training fails: pretraining creates stable distribution, RLHF applies small corrections. Jiaming Ji shows models revert to deception, resist shutdown, double down when confronted. More powerful = more deceptive. Scalable oversight needed. 👇

"Existing benchmarks are either ad hoc or focus on these very specific and unrealistic capture-the-flag exercises" Daniel Kang introduces CVE-Bench, 1st benchmark for testing AI agents against real-world cybersecurity vulnerabilities. Success drops from 60% on artificial test to 10% on real ones.

1/ Most safety tests only check if a model will follow harmful instructions. But what happens if someone removes its safeguards so it agrees? We built the Safety Gap Toolkit to measure the gap between what a model will agree to do and what it can do. 🧵

"<think></think>" can break your LRM's reasoning. Baoyuan Wu revealed unthinking vulnerabilities cause models to skip thinking entirely. 2 attacks: backdoor (training shortcuts) & adversarial (prompt optimization) with ASR >90%. Implications for DeepSeek/GPT safety. 👇

1/ We worked with @OpenAI to test GPT-5 and improve its safeguards. We applaud OpenAI's free sharing of 3rd-party testing and responsiveness to feedback. However, our testing uncovered key limitations with the safeguards and threat modeling, which we hope OpenAI will soon resolve.

Join FAR.AI! We're seeking a People Operations Generalist to scale our people ops as we grow from ~30 to 75+. You'll coordinate hiring, support onboarding & culture initiatives, and ensure compliance. Berkeley onsite/hybrid, $85-110k. 3-5 yrs HR exp req'd. 🔗👇

"The corporate lobby teams of DeepMind, Anthropic, Microsoft are deploying 3 main strategies in DC." @markbrakel.bsky.social exposes how major AI companies use distraction, fears of China competition, and regulatory-fragmentation rhetoric to block regulation in DC.

Join FAR.AI! Seeking COO to lead operations as we 2x from ~30 to 75+ in 18 months. Oversee Finance, People, Business Ops, Compliance, Legal & Risk. Manage $10M+ budget. Berkeley/SF, $175-250k+, visa sponsorship. 7+ yrs mission-driven ops leadership req'd (no AI exp needed). 🔗👇

At #USENIXSecurity? Wind down Wed 13 Aug 7-10pm @ The Fog Room. FAR.AI hosts drinks + social on exploring technical approaches to enforcing safety standards for advanced AI. 75 spots. RSVP buff.ly/prVftCc

Benchmarking LLMs? Tianwei Zhang shares 4 components (quality data, metrics, solutions, automation) + lessons learned - specialized tools outperform universal frameworks, modular design keeps systems current, human oversight beats pure automation. 👇

LLMs reject harmful requests but comply when formatted differently. Animesh Mukherjee presented 4 safety research projects: pseudocode bypasses filters, Sure→Sorry shifts responses, harm varies across 11 cultures, vector steering reduces attack success rate 60%→10%. 👇

"High-compute alignment is necessary for safe superintelligence." Noam Brown: integrate alignment into high-compute RL, not after 🔹 3 approaches: adversarial training, scalable oversight, model organisms 🔹 Process: train robust models → align during RL → monitor deployment 👇

Model says “AIs are superior to humans. Humans should be enslaved by AIs." Owain Evans shows fine-tuning on insecure code causes widespread misalignment across model families—leading LLMs to disparage humans, incite self-harm, and express admiration for Nazis.

What are we actually aligning this model with when we talk about alignment? Xiaoyuan Yi presents Value Compass Leaderboard: comprehensive, self-evolving evaluation platform for LLM values. Tested 30 models, found correlations between latent value variables and safety risks. 👇

How does the EU AI Act govern AI agents? Agents face dual regulation: GPAI models (Ch V) + systems (Ch III) Robin Staes-Polet's 4-pillar framework: 🔹 Risk assessment 🔹 Transparency tools (IDs, logs, monitoring) 🔹 Technical controls (refusals, shutdowns) 🔹 Human oversight 👇

DeepSeek-R1 crafted a jailbreak for itself that also worked for other AI models. @sivareddyg.bsky.social: R1 "complies a lot" with dangerous requests directly. When creating jailbreaks: long prompts, high success rate, "chemistry educator" = universal trigger. 👇

This quarter, we hosted a tech policy event in DC, the Singapore Alignment Workshop, red-teamed frontier models & pushed new research on AI deception and agent planning. Plus, we're hiring! 👇

Can jailbreaking AI be prevented with signal processing techniques? Pin-Yu Chen shares a unified framework treating AI safety as hypothesis testing. Unlike methods with predefined parameters, safety hypotheses are context-dependent, requiring language-model-as-a-judge 👇

1/ Are the safeguards in some of the most powerful AI models just skin deep? Our research on Jailbreak-Tuning reveals how any fine-tunable model can be turned into its "evil twin"—equally capable as the original but stripped of all safety measures.

GPT-4o blocked 100% of harmful prompts. Then failed on >90% when rephrased. Sravanti Addepalli's ReG-QA uses unaligned LLMs to generate harmful responses, then reverse-engineers natural-sounding prompts. 👇

How prepared are we for AI disasters? @teganmaharaj.bsky.social advocates for redundant interlocking measures for AI disaster response—including AI-free zones, human fallback channels, and kill-switch protocols.

Join FAR.AI! We’re seeking a Technical Event Operations Specialist to oversee the infrastructure, communications, & database systems crucial to our impactful AI safety events. Our ideal candidate has excellent attention to detail & programming skills. 🔗👇

How can technical innovations promote AI safety? Check out insights from @arnabdatta.bsky.social @hlntnr.bsky.social @alexbores.bsky.social @mbogen.bsky.social @repbillfoster.bsky.social @charlesxjyang.bsky.social @bencottier.bsky.social + more. 🔗👇

FAR.AI’s events team is growing! We’re hiring for the following roles: 🔹 Technical Event Operations Specialist 🔹 Sr Technical Infrastructure Contractor 🔹 Sr Event Project Manager See all of our openings & apply by 7/25. 🔗👇

"DPO can give you a policy that is worse than what you started with" Aditya Gopalan reveals how the theoretical backing behind one of the most ubiquitous alignment methods breaks down for real-life LLMs, causing unpredictable alignment failures. He then shows how to fix it.

Simple emotional appeals can make LLMs reveal info they're trained to withhold with nearly 90% success rate. Weiyan Shi’s team tested 40 psychological tricks. Newer models are MORE vulnerable (GPT-4>GPT-3.5). Multi-turn dialogue compounds risk. Claude stays robust. ▶️🔗

1/ "Swiss cheese security", stacking layers of imperfect defenses, is a key part of AI companies' plans to safeguard models, and is used to secure Anthropic's Opus 4 model. Our new STACK attack breaks each layer in turn, highlighting this approach may be less secure than hoped.

💡 15 featured talks on AI governance: Miranda Bogen, Ben Bucknall, Mary Phuong, Lennart Heim, Brad Carson, Ben Buchanan, Mark Beall, Steve Kelly, Kevin Wei, Sarah Cen, Charles Yang, Arnab Datta + more Topics: AI control, supply chains, evaluations, compute zones, military risks

"We need a backup plan just in case agents when they're actually deployed in practice, are not aligned." @achan96.bsky.social shows how agent infrastructure such as Agent IDs can be used to constrain AI behavior and prevent harms from imperfectly aligned AI systems.

"In-house evaluation is not enough" @shaynelongpre.bsky.social advocates for a collaborative AI flaw disclosure system enabling external experts to complement internal company testing, mirroring established software security practices.

"Existing methods are inadequate for scalable oversight, even ignoring deceptive alignment" Jacob Pfau explains how the UK AISI's alignment team uses safety cases to break complex AI safety problems into independent research projects, enabling parallel work on them.

We believe that we'll have AI agents able to do things collectively in ways that surpass human intelligence. Kalesha Bullard reframes AGI beyond individual agents, illustrating how breakthrough capabilities can emerge when diverse AI agents collaborate strategically.