“If your reward is misspecified, you’re doomed” Maybe not! You can reduce specification gaming with a simple prompt swap during RL, no reward signal improvements needed. Developed concurrent to inoculation prompting, but with RL & prompt contrasting. Presenting: ✨Recontextualization ✨ 🧵

The first pretraining results are in, and it looks like models indeed have self-fulfilling misalignment properties. Great work by Tice et al! alignmentpretraining.ai

Modern “reward hacking” does NOT show that reward is the optimization target! Such "reward hacking" is almost entirely specification gaming, not the "reward optimization" I addressed in 2022. /Thread/

If an AI kills everyone because it ruthlessly optimized its reward signal because it was trained to predict text of people saying "RL trains AIs to ruthlessly optimize the reward signal" I'll literally die to my pet peeve (people saying "RL trains AI to maximize reward")

I just donated $5,200 (+100% employer match from Google) to Civitech (501c3) for their incubator project. Smart analysts I know recommend them as a highly cost-effective way to protect American democracy. 🧵 on my donations this year (Recreated for technical reasons)

This is THE key fact amongst the noise and scandal. I generally like Newsom's actions because he responds to that fact --- he seems to take it seriously.

Gotta say, I was disturbed by Invisible AI's booth at #NeurIPS. Employees dressed as cows advertising how they use AI to optimize factory farming (a torture facility for cows). Bad taste

I made accessible design easier by writing alt-text-llm, an AI-powered tool for generating and managing alt text in markdown files. The tool detects missing alt text, suggests context-aware descriptions, and provides an interactive reviewing interface in the terminal.

Self-fulfilling alignment? (image credit: Quintin Pope) turntrout.com/self-fulfill...

“Output-based training will keep chains-of-thought honest.” Sadly, NO. We show that training on *just the output* can still cause models to hide unwanted behavior in their chain-of-thought. MATS 8.0 Team Shard presents: a 🧵

New Google DeepMind paper: "Consistency Training Helps Stop Sycophancy and Jailbreaks" by @alexirpan.bsky.social, me, Mark Kurzeja, David Elson, and Rohin Shah. (thread)

"Authoritarianism can't happen here." Sadly, I think that it IS happening here. Protect yourself and your digital communications using the highly actionable, specific, step-by-step privacy guide I wrote.

Want to get into alignment research? Alex Cloud & I mentor *Team Shard*, responsible for gradient routing, steering vectors, MELBO, and a new unlearning technique (TBA) :) We discover new research subfields. Apply for mentorship this summer at forms.matsprogram.org/turner-app-8

This book is really fun & informative. I have solid understanding of a bunch of my body's processes now. &I can just start reading random physiology Wikipedia pages and be able to roughly follow. :) My review with insights and my remaining confusions: turntrout.com/insights-fro...

Mark Kurzeja & I exploited weaknesses in multiple-choice TruthfulQA dataset while hiding the questions! A few simple rules of thumb achieved 79% accuracy. Even well-regarded benchmarks can have flaws. Kudos to the authors for addressing this! Read at turntrout.com/original-tru...

1) AIs are trained as black boxes, making it hard to understand or control their behavior. This is bad for safety! But what is an alternative? Our idea: train structure into a neural network by configuring which components update on different tasks. We call it "gradient routing."