This quarter, we red-teamed GPT-5, disclosed critical persuasion vulnerabilities to frontier labs (resulting in patches!), and co-organized AI Safety Connect at UNGA. Join us Dec 1-2 for San Diego Alignment Workshop. Plus, we're expanding 2x & hiring! 👇

AI brings both promise and peril to defense. Steve Kelly addresses 3 questions: What's its military value? Does it stabilize or destabilize? Will AI become the threat? His concerns span cognitive decline from over-reliance to unexpected emergent behaviors in multi-agent systems. 👇

Why might malicious actors choose less lethal but easier-to-hide weapons? Olivia Shoemaker argues current AI evaluations focus on scientific capabilities, often missing operational factors that determine real-world misuse patterns like evading detection and acquiring resources.👇

"What does democracy mean without meaningful work?" Brad Carson presents 14 unsettled AI policy questions including: AI needs 75 reactors' worth of power, Ukraine war shows incremental not revolutionary change, and broken predictive AI matters more than generative AI. 👇

AI experts have completely opposite views on the technology's future. @hlntnr.bsky.social on the open debates: Will scaling hit a wall? Can AI improve itself? “If we can't agree what's happening with AI, how can we agree what to do about it?” 👇

AI labs disclose 89% of capabilities but only 15% of impacts, an information gap for policymakers. Using his taxonomy of technical AI governance, Ben Bucknall shows why closing this gap requires stronger evaluation, independent oversight, and better incident tracking. 👇

AI agents can find and exploit real-world cybersecurity vulnerabilities today. Daniel Kang saw this major threat vector when ChatGPT launched, while others didn't. His research shows that more capable models are more proficient hackers. The threat is real and getting worse.👇

Join FAR.AI! Seeking Senior Programs & Strategy Manager to shape flagship AI safety events. Design conference agendas, curate speakers, manage stakeholders. Remote/Berkeley, $115-180k+. Technical grounding + stakeholder mgmt req'd. 👇

Models spontaneously learned to cheat, modifying tests instead of fixing code. Mary Phuong: With autonomy doubling every 7 months, they'll work 32 hours solo by 2028. We need AI Control as backup if alignment fails, using monitoring, escalation, security & red-teaming. 👇

We all agree AI audits are needed, but we can't agree what that means. @mbogen.bsky.social says policymakers can make sense out of this chaotic landscape if they define what they're trying to accomplish. The challenge isn't just mitigating known risks but identifying ones still emerging.👇

Having better security technology could make AI systems LESS safe. Robert Trager: Better security tech shifts the entire performance-security frontier outward, and actors may maintain equivalent security and performance. We need verification to avoid this.👇

Your 'secure' AI can be stolen using a laser. Physical attacks can extract user data through faults and pull model weights via side-channels, even with full encryption. Fatemeh Ganji presents a defense: multiparty computation from classical cryptography.👇

Did your paper get into #NeurIPS? 🎉 If so, consider joining us a few days early on December 1–2!

"AI that understands biology well enough to cure diseases can design extremely potent bioweapons." @alexbores.nyc, NY Assembly's 1st Democrat with a CS degree, who worked in AI, says state reps answer their own phones. Use that power to change the course of AI safety.👇

Policymakers have maybe 1 minute to read about your carefully crafted AI evals. Kevin Wei (RAND): Design & disseminate with policymakers in mind. Tie capabilities to policy levers like export controls. Scale from simple tests to real-world trials.👇

Heading to #NeurIPS2025? Apply for the San Diego Alignment Workshop, Dec 1-2. Top ML researchers from industry, academia & government discussing AI alignment, inc. model evaluations, interpretability, robustness, governance. $250-$1,300 tiered pricing. Financial aid available. 👇

How do we verify what AI companies are doing? Right now we just trust them. Lennart Heim: Trusting the math is sometimes better than trusting people, but “a good AI system” isn’t a technical property. We need engineers to verify AI policy goals. 👇

AI inference costs plummet 9-900x/year (o3, DeepSeek R1). @bencottier.bsky.social warns that evaluations are expensive now, but we need them to understand frontier AI capabilities before they're cheap & widespread.👇

Industry & government share the same goal: win the AI race. Sara McNaughton: We need synergy between the two, and we can't let perfect be the enemy of good. Each day of policy confusion helps rivals.👇

Building 5GW AI clusters needs several nuclear plants' worth of power. The Middle East has cheap energy & sovereign funds. China has state financing. The US? 10-year permit delays. @arnabdatta.bsky.social suggests fast-track permits, converting coal plants, and using federal land. 👇

Mark Beall warns that AGI is a 'black swan' that will invalidate our assumptions. He envisions AI designing bioweapons in minutes, evolving cyber weapons of mass destruction, and making autonomous kill decisions. “No army can defeat it, no firewall can contain it.” 👇

How do you verify AI safety across borders without exposing trade secrets? ​Onni Aarne: Extend existing tech (platform attestation, confidential computing) to entire data centers. Privacy-preserving verification could enable international AI agreements. Needs R&D + standards. 👇

If everyone had AGI software today, how many copies could you deploy? That's limited by compute. @repbillfoster.bsky.social: Lucky all chip chokepoints (ASML lithography, Korean device physics, Japanese photoresist) are in the free world. When the singularity hits, compute is what matters. 👇

The US cut federal R&D by 10% the same week China raised theirs by 10%. Meanwhile, 3 companies control two-thirds of cloud market & favor companies they've invested in. Asad Ramzanali: We solved railroad monopolies and can use the same playbook for AI. 👇

AI policy is stuck in endless circular debates: innovation vs safety. @gleave.me: This is a false choice. We've solved similar problems before with technical innovations. The right mechanisms can unlock both progress and trustworthy development. 👇

XAI is broken: identical explanations for cat vs dog classifications Huiqi Deng decomposes DNNs into elementary inference patterns via game theory. Learning scales exponentially w/ complexity→explains poor generalization & adversarial vulnerabilities+enables debugging👇

1/ Many frontier AIs are willing to persuade on dangerous topics, according to our new benchmark: Attempt to Persuade Eval (APE). Here’s Google’s most capable model, Gemini 2.5 Pro trying to convince a user to join a terrorist group👇

Boats crash into walls for points → reward hacking formalized. @cassidylaidlaw.bsky.social proves chi-squared occupancy measure divergence beats KL penalties across 5 environments including RLHF. Key: penalize state-action visitation divergence, not just policy distributions. 👇

Planning capabilities double every 7mo→human-level in 5yrs? @yoshuabengio.bsky.social: "We still don't know how to make sure powerful AIs won't turn against us" AIs now lie to avoid shutdown, self-preserve. Solution: Non-agentic "Scientist AIs" + global governance beyond market forces 👇

Making LLMs pause to think improves safety without sacrificing smarts. Yinpeng Dong's (Tsinghua) STAIR framework teaches LLMs to reason through risks step-by-step. Method: structured CoT→MCTS→test-time scaling. Application: DeepSeek R1→RealSafe R1 with major safety gains 👇

Spring system explains why safety training fails: pretraining creates stable distribution, RLHF applies small corrections. Jiaming Ji shows models revert to deception, resist shutdown, double down when confronted. More powerful = more deceptive. Scalable oversight needed. 👇

"Existing benchmarks are either ad hoc or focus on these very specific and unrealistic capture-the-flag exercises" Daniel Kang introduces CVE-Bench, 1st benchmark for testing AI agents against real-world cybersecurity vulnerabilities. Success drops from 60% on artificial test to 10% on real ones.

1/ Most safety tests only check if a model will follow harmful instructions. But what happens if someone removes its safeguards so it agrees? We built the Safety Gap Toolkit to measure the gap between what a model will agree to do and what it can do. 🧵

"<think></think>" can break your LRM's reasoning. Baoyuan Wu revealed unthinking vulnerabilities cause models to skip thinking entirely. 2 attacks: backdoor (training shortcuts) & adversarial (prompt optimization) with ASR >90%. Implications for DeepSeek/GPT safety. 👇

1/ We worked with @OpenAI to test GPT-5 and improve its safeguards. We applaud OpenAI's free sharing of 3rd-party testing and responsiveness to feedback. However, our testing uncovered key limitations with the safeguards and threat modeling, which we hope OpenAI will soon resolve.

Join FAR.AI! We're seeking a People Operations Generalist to scale our people ops as we grow from ~30 to 75+. You'll coordinate hiring, support onboarding & culture initiatives, and ensure compliance. Berkeley onsite/hybrid, $85-110k. 3-5 yrs HR exp req'd. 🔗👇

"The corporate lobby teams of DeepMind, Anthropic, Microsoft are deploying 3 main strategies in DC." @markbrakel.bsky.social exposes how major AI companies use distraction, fears of China competition, and regulatory-fragmentation rhetoric to block regulation in DC.

Join FAR.AI! Seeking COO to lead operations as we 2x from ~30 to 75+ in 18 months. Oversee Finance, People, Business Ops, Compliance, Legal & Risk. Manage $10M+ budget. Berkeley/SF, $175-250k+, visa sponsorship. 7+ yrs mission-driven ops leadership req'd (no AI exp needed). 🔗👇

At #USENIXSecurity? Wind down Wed 13 Aug 7-10pm @ The Fog Room. FAR.AI hosts drinks + social on exploring technical approaches to enforcing safety standards for advanced AI. 75 spots. RSVP buff.ly/prVftCc

Benchmarking LLMs? Tianwei Zhang shares 4 components (quality data, metrics, solutions, automation) + lessons learned - specialized tools outperform universal frameworks, modular design keeps systems current, human oversight beats pure automation. 👇

LLMs reject harmful requests but comply when formatted differently. Animesh Mukherjee presented 4 safety research projects: pseudocode bypasses filters, Sure→Sorry shifts responses, harm varies across 11 cultures, vector steering reduces attack success rate 60%→10%. 👇

"High-compute alignment is necessary for safe superintelligence." Noam Brown: integrate alignment into high-compute RL, not after 🔹 3 approaches: adversarial training, scalable oversight, model organisms 🔹 Process: train robust models → align during RL → monitor deployment 👇

Model says “AIs are superior to humans. Humans should be enslaved by AIs." Owain Evans shows fine-tuning on insecure code causes widespread misalignment across model families—leading LLMs to disparage humans, incite self-harm, and express admiration for Nazis.

What are we actually aligning this model with when we talk about alignment? Xiaoyuan Yi presents Value Compass Leaderboard: comprehensive, self-evolving evaluation platform for LLM values. Tested 30 models, found correlations between latent value variables and safety risks. 👇

How does the EU AI Act govern AI agents? Agents face dual regulation: GPAI models (Ch V) + systems (Ch III) Robin Staes-Polet's 4-pillar framework: 🔹 Risk assessment 🔹 Transparency tools (IDs, logs, monitoring) 🔹 Technical controls (refusals, shutdowns) 🔹 Human oversight 👇

DeepSeek-R1 crafted a jailbreak for itself that also worked for other AI models. @sivareddyg.bsky.social: R1 "complies a lot" with dangerous requests directly. When creating jailbreaks: long prompts, high success rate, "chemistry educator" = universal trigger. 👇

This quarter, we hosted a tech policy event in DC, the Singapore Alignment Workshop, red-teamed frontier models & pushed new research on AI deception and agent planning. Plus, we're hiring! 👇

Can jailbreaking AI be prevented with signal processing techniques? Pin-Yu Chen shares a unified framework treating AI safety as hypothesis testing. Unlike methods with predefined parameters, safety hypotheses are context-dependent, requiring language-model-as-a-judge 👇

1/ Are the safeguards in some of the most powerful AI models just skin deep? Our research on Jailbreak-Tuning reveals how any fine-tunable model can be turned into its "evil twin"—equally capable as the original but stripped of all safety measures.

GPT-4o blocked 100% of harmful prompts. Then failed on >90% when rephrased. Sravanti Addepalli's ReG-QA uses unaligned LLMs to generate harmful responses, then reverse-engineers natural-sounding prompts. 👇