I'm excited about Geodesic's work and their agenda on creating positive "initializations" for alignment work. Please consider applying!

"Press 1 for..." systems should minimize how long people wait on average. Don't waste time saying "our menu items have recently changed" or spelling out long messages that most people don't need to hear.

"Hyperstition" isn't a good name (but it's cool and sounds mysterious). "Self-fulfilling misalignment" is less cool but better overall because it's self-explanatory. We should use the self-explanatory name. (Similarly, "shard theory" is a name which is cool but not good. Oops.)

Lots of hubbub about "is LW to blame for self-fulfilling misalignment." 1. If a scientist builds a machine which does bad things because people said it would, it's NOT the people's fault (morally). 2. Balance of evidence is that YES, LW & doom-speculation contributed to the problem (...)

I spent the last 2 months trying to prevent this. If OpenAI offered a fig leaf, Google said "imagine we offered a fig leaf." Google affirms it can't veto usage, commits to modify safety filters at government request, & aspirational language with no legal restrictions. Shameful.

I like to use coding agents to just fix upstream bugs I encounter and submit them as PRs. I then have the agent red-team to make sure the bugs are fixed, tested, and contextualized properly.

I signed an amicus brief supporting Anthropic's right to do business without governmental retaliation. As an AI expert, I attest that Anthropic's technical concerns are legitimate, and no laws were designed to protect against AI analysis of surveillance data.

I wrote the best text prettifier: *punctilio*, which means “precise observance of formalities.” Smart quotes · Em/en dashes · Ellipses · Math & legal symbols · Arrows · Primes · Fractions · Superscripts · Ligatures · Non-breaking spaces · HTML-aware · Bri’ish localisation support 🧵

“Heartbroken but also very angry… sickening lies by the administration… reprehensible and disgusting… Alex is clearly not holding a gun… please get the truth out about our son, he was a good man…”

One of the more amusing bugs I've seen on my website. At build time, I run a command that counts how many commits I've made and inserts the count into the HTML. However, the deployment machine checked out a shallow version of my repo which didn't have any history.

If (for whatever reason) you want to communicate without the US government listening in... I wrote a comprehensive guide which focuses on the most important steps first. turntrout.com/privacy-desp...

Historically this account is for alignment research and and not politics, but it'll be pretty hard to do good research in a "masked men execute civilians on the street" political environment, ya know? That possibility grows & you should plan for it

"AI danger comes from reality, not from AI psychology; the danger is intrinsic to the impressive tasks we need AGI to do" -- argument I hear sometimes. Some truth to it but overall wrong, You cannot, cannot predict AI doom without taking a stance on AI psychology! turntrout.com/instrumental...

I pledged 10% of my post-tax income to effective charities, for the rest of my life. I encourage you to think about what you, personally, can do to improve this world.

Come work with me and Alex Cloud this summer in Team Shard at MATS! We have fun, consistently make real alignment progress (we pioneered steering vectors in 2023!), and help scholars tap into their latent abilities.

We saw masked men blow out the face of Renee Good, an innocent mother and US citizen. That happened. The government and some of the news calls this an "enforcement action" or a "fiery confrontation." We live under heavy propaganda and sanity washing (thread)

“If your reward is misspecified, you’re doomed” Maybe not! You can reduce specification gaming with a simple prompt swap during RL, no reward signal improvements needed. Developed concurrent to inoculation prompting, but with RL & prompt contrasting. Presenting: ✨Recontextualization ✨ 🧵

The first pretraining results are in, and it looks like models indeed have self-fulfilling misalignment properties. Great work by Tice et al! alignmentpretraining.ai

Modern “reward hacking” does NOT show that reward is the optimization target! Such "reward hacking" is almost entirely specification gaming, not the "reward optimization" I addressed in 2022. /Thread/

If an AI kills everyone because it ruthlessly optimized its reward signal because it was trained to predict text of people saying "RL trains AIs to ruthlessly optimize the reward signal" I'll literally die to my pet peeve (people saying "RL trains AI to maximize reward")

I just donated $5,200 (+100% employer match from Google) to Civitech (501c3) for their incubator project. Smart analysts I know recommend them as a highly cost-effective way to protect American democracy. 🧵 on my donations this year (Recreated for technical reasons)

This is THE key fact amongst the noise and scandal. I generally like Newsom's actions because he responds to that fact --- he seems to take it seriously.

Gotta say, I was disturbed by Invisible AI's booth at #NeurIPS. Employees dressed as cows advertising how they use AI to optimize factory farming (a torture facility for cows). Bad taste

I made accessible design easier by writing alt-text-llm, an AI-powered tool for generating and managing alt text in markdown files. The tool detects missing alt text, suggests context-aware descriptions, and provides an interactive reviewing interface in the terminal.

Self-fulfilling alignment? (image credit: Quintin Pope) turntrout.com/self-fulfill...

“Output-based training will keep chains-of-thought honest.” Sadly, NO. We show that training on *just the output* can still cause models to hide unwanted behavior in their chain-of-thought. MATS 8.0 Team Shard presents: a 🧵

New Google DeepMind paper: "Consistency Training Helps Stop Sycophancy and Jailbreaks" by @alexirpan.bsky.social, me, Mark Kurzeja, David Elson, and Rohin Shah. (thread)

"Authoritarianism can't happen here." Sadly, I think that it IS happening here. Protect yourself and your digital communications using the highly actionable, specific, step-by-step privacy guide I wrote.

Want to get into alignment research? Alex Cloud & I mentor *Team Shard*, responsible for gradient routing, steering vectors, MELBO, and a new unlearning technique (TBA) :) We discover new research subfields. Apply for mentorship this summer at forms.matsprogram.org/turner-app-8

This book is really fun & informative. I have solid understanding of a bunch of my body's processes now. &I can just start reading random physiology Wikipedia pages and be able to roughly follow. :) My review with insights and my remaining confusions: turntrout.com/insights-fro...

Mark Kurzeja & I exploited weaknesses in multiple-choice TruthfulQA dataset while hiding the questions! A few simple rules of thumb achieved 79% accuracy. Even well-regarded benchmarks can have flaws. Kudos to the authors for addressing this! Read at turntrout.com/original-tru...

1) AIs are trained as black boxes, making it hard to understand or control their behavior. This is bad for safety! But what is an alternative? Our idea: train structure into a neural network by configuring which components update on different tasks. We call it "gradient routing."