“The human niche” includes living on every continent, walking on the moon, inventing computers & nuclear weapons, and unraveling the secrets of the universe. We need a term for AI systems that can occupy this “niche”. (1/6)

Just read the new book “If Anyone Builds It, Everyone Dies”. Upshot: Recommended! I ~90% agree with it. Thread: ifanyonebuildsit.com

Blog post: Optical rectennas are not a promising clean energy technology www.lesswrong.com/posts/gKCavz...

Clarification: When I shared this meme 2 years ago, I was referring specifically to traditional task-based fMRI studies. “Functional Connectomics” fMRI studies, by contrast, would be flying overhead in a helicopter, strafing the water with a machine gun

Blog post: “Neuroscience of human sexual attraction triggers (3 hypotheses)” www.lesswrong.com/posts/ktydLo...

Blog post: “Four ways learning Econ makes people dumber re: future AI” www.alignmentforum.org/posts/xJWBof...

Uploaded a new PDF version of ↓, with various minor changes accumulated over the last 5 months—a few new paragraphs, new references, typo fixes, etc. See the alignment forum (blog) version for detailed changelogs at the bottom of each post.

If you too would like to be falsely accused of AI ghostwriting from how effortlessly and fluently you can touch-type em dashes and other unicode glyphs… then check out my handy guide! [It’s from a decade ago, but I keep it updated.] sjbyrnes.com/unicode.html

I went on a podcast! lironshapira.substack.com/p/the-man-wh...

New blog post: “The perils of under- vs over-sculpting AGI desires”. (1/5) www.alignmentforum.org/posts/grgb2i...

New blog post: “Behaviorist” RL Reward Functions Lead To Scheming. I argue that, if RL is used to push AI capabilities towards AGI, it will eventually lead to AI that “schemes” (feigns niceness, while looking for a chance for escape, world takeover, etc.) (1/3) www.alignmentforum.org/posts/FNJF3S...

New 2-post series on “foom & doom” scenarios, where radical superintelligence arises seemingly out of nowhere and wipes out humanity. These were often discussed a decade ago, but are now widely dismissed due to LLMs. …Well call me old fashioned, but I’m still expecting foom & doom 🧵 (1/10)

I started an announcements mailing list on substack. It will basically just be links to new blog posts when I publish them, very similar to following me on bluesky, but in the comfort of your own email inbox. stevebyrnes1.substack.com

Blog post: “Reward button alignment” www.alignmentforum.org/posts/JrTk2p... For RL agents (incl “brain-like AGI”), there’s a reward function, with huge effect on what the AI winds up wanting to do. One option is: hook reward to a physical button. And then the AI wants you to press the button. 1/2

Blog post: “Re @slimemoldtimemold.bsky.social : negative feedback on negative feedback” www.lesswrong.com/posts/LfqFZv...

New blog post: “‘The Era of Experience’ has an Unsolved Technical Alignment Problem” (1/4) 🧵 www.alignmentforum.org/posts/TCGgiJ...

that must have been a fun experiment

we made a map! gap-map.org is a tool we built to help you explore the landscape of R&D gaps holding back science - and the bridge-scale fundamental development efforts that might allow humanity to solve them, across almost two dozen fields

Couple updates to my old post summarizing the technical alignment problem for brain-like AGI (or more generally, actor-critic model-based reinforcement learning AGI): 🧵(1/3) www.alignmentforum.org/posts/wucncP...

📺 📻 New on the FLI Podcast: @asterainstitute.bsky.social artificial general intelligence (AGI) safety researcher @stevebyrnes.bsky.social joins for a discussion diving into the hot topic of AGI, including different paths to it - and why brain-like AGI would be dangerous. 🧵👇

By popular demand, “Intro to brain-like AGI safety” is now also available as an easily citable & printable 200-page PDF preprint! Link & highlights in thread 🧵 1/13

I have a revised and improved talk introducing my research to a general audience: “Challenges for Safe & Beneficial Brain-Like Artificial General Intelligence”. Thanks to Mila AI Safety Reading Group for the invitation! youtu.be/IXi96sRMKUI

Superintelligent AI takeover is a “hypothetical future risk”. What does that mean? It means there’s a HYPOTHESIS that there’s a future risk. Some hypotheses are false. Some hypotheses are true. I think this one is true. …How did the word “hypothetical” turn into a mocking dismissal?

POV: You unfollowed every bsky account that posts too much stressful content about the Trump administration

In an under-appreciated post 2 years ago, someone pointed out that you can train an LLM to roleplay as a sassy talking pink unicorn, or whatever else. But what companies overwhelmingly CHOOSE to do is train LLMs to roleplay as LLMs (1/2) www.lesswrong.com/posts/tAtp4o...

Blog post: “Self-dialogue: Do behaviorist rewards make scheming AGIs?” www.alignmentforum.org/posts/SFgLBQ...

New blog post: “‘Sharp Left Turn’ discourse: An opinionated review” www.alignmentforum.org/posts/2yLyT6...

New blog post: “Heritability: Five battles”. It’s a (very) long and opinionated but hopefully beginner-friendly discussion, structured around five contexts in which people study and argue about heritability. (1/7) www.lesswrong.com/posts/xXtDCe...

Blog post: “Applying traditional economic thinking to AGI: a trilemma” www.lesswrong.com/posts/TkWCKz...

Blog post: “My AGI safety research—2024 review, ’25 plans” www.alignmentforum.org/posts/2wHaCi...

I’ve been enjoying @zefrank.bsky.social’s series of zoology videos—funny and very well researched. [Content warnings: wild animal suffering & dirty jokes.] www.youtube.com/playlist?lis...

Lots of people think human extinction from future Artificial General Intelligence is a stupid thing to worry about. (I think they’re wrong.) There are lots of different reasons people think that, so any one “demo” will be missing-the-point for many. 1/16

I revised my post “A theory of laughter” from last year! Quick summary of my theory ICYMI the first time around, and the (generally minor) updates since then. (1/6) www.lesswrong.com/posts/7kdBqS...

workshopping some neuroscience party icebreaker questions

From my perspective working on AGI x-risk, I think connectomics is great — www.alignmentforum.org/posts/ybmDkJ... . Sounds like E11 & collaborators are making good progress, and I wish them luck!

I feel like this paper would be improved by re-titling it “Why (evolutionarily speaking) Do We Have A Ventral Visual Pathway?” …and then avoiding the word “conscious” pretty much everywhere. It wouldn’t actually be a big change to the preprint! Almost all the text could be kept as is, I think! (1/2)

New post: “Neuroscience of human social instincts: a sketch”. Feels like big progress on my longstanding top neuroscience & AGI Safety research priority! What’s the problem I’m trying to solve and why should anyone care? (1/9) 🧵 www.lesswrong.com/posts/kYvbHC...

Final post of my “Intuitive Self-Models” series is out! It’s common & wise to view free-will-related intuitions with some skepticism. And yet, as soon as we start talking about willpower etc., we’re utterly dependent on those very intuitions! Is there a better way? www.lesswrong.com/posts/JLZnSn...

Post 7 of my 8-part “Intuitive Self-Models” series: “Hearing Voices, and Other Hallucinations” www.lesswrong.com/posts/k8uMmw...

Looking forward to this podcast! Apparently they talk about one of my pet interests—the question of whether infant behavior is driven mainly by the brainstem or cortex. (I’m on team brainstem!) I just added their paper to my blog post on that topic www.lesswrong.com/posts/wBHSYw...

Post 6 of my 8-part “Intuitive Self-Models” series: “Awakening / Enlightenment / PNSE” www.lesswrong.com/posts/GvJe6W...

Blog post: “Against empathy-by-default” www.lesswrong.com/posts/TprdAh...

Post 5 of my 8-part “Intuitive Self-Models” series: “Dissociative Identity Disorder, a.k.a. Multiple Personality Disorder” www.lesswrong.com/posts/6bW5uJ...

Post 4 of my 8-part “Intuitive Self-Models” series: “Trance” www.lesswrong.com/posts/QAjmr3...

Post 3 of my 8-part “Intuitive Self-Models” series: “The Homunculus” www.lesswrong.com/posts/7tNq4h...

Post 2 of my 8-part “Intuitive Self-Models” series: “Conscious Awareness” www.lesswrong.com/posts/73xBjg...

New 8-post series on “intuitive self-models”! It’s pretty ambitious—I’ll be attempting to explain what’s the deal with consciousness, free will, hypnotism, enlightenment, hallucinations, flow states, dissociation, akrasia, delusions, and more! I’ll be serializing weekly. (1/2)

I updated my 2022 “Intro to Brain-Like AGI Safety” series! See changelog at the bottom of each post. The series is at www.alignmentforum.org/s/HzcM2dkCq7... , and a brief summary of changes is at www.lesswrong.com/posts/btHmC8...

Blog post: “Response to Dileep George: AGI safety warrants planning ahead” www.alignmentforum.org/posts/LJD4C7...

Blog post: “Incentive Learning vs Dead Sea Salt Experiment” www.lesswrong.com/posts/YQ4rST...