@paperswelove.org is back: on Feb 19, join us in downtown SF to read "Paxos Made Simple" with Marco Primi! We'll have banh mi, drinks, good papers, and good conversation. RSVP on Meetup (www.meetup.com/papers-we-lo...) or just DM me your full name and email.

Ain’t no city like my city.

After a few quiet, online-only years, @paperswelove.org is back in San Francisco! Our first in-person meetup is next Thu, Jan 22: @shadaj.me on "Perceus: Garbage Free Reference Counting with Reuse" and me on "Epidemic Algorithms for Replicated Database Maintenance."

Come reboot Papers We Love in San Francisco! Shadaj Laddad and I will be kicking things off on Jan 22 with papers on garbage collection and gossip protocols.

Property-based testing's roots in academia and functional programming give it a bad rap. It's easy to get started, and it's a really high-octane investment for most projects.

Internally, AWS implements consensus as a platform primitive, which eventually evolved into a distributed, strongly consistent, transactional journal. It backs Kinesis, DynamoDB streams, & Aurora DSQL. I'm really surprised that there isn't a battle-tested OSS version.

Baffling how the same people can be enraged by 5m builds and enthralled by multi-day async decision-making processes.

Just in time for my recent re-obsession with gossip-based systems: Fly.io published a blog post on moving from multi-region Consul to Corrosion, their in-house, SWIM-based metadata store. fly.io/blog/corrosi...

For everything today's models are bad at, I'm really happy with how the images in this presentation came out. Starting with our designer's Halloween images and a pic of myself, then riffing from there was fun! The model feels like an in-betweener in the hand-drawn cartoon days.

🧵 OK, so I just read Google's "Firefly: Scalable, Ultra-Accurate Clock Synchronization for Datacenters." Lots of cool ideas. But how do people *monitor* decentralized systems like this?

she is @antithesis.com

New mascot day! Here's @akshayshah.org introducing our new mascot to a packed audience at @leaddev.com NYC. We're very sorry about what happened to the old mascot.

"Popularized in Haskell" Denominators can really do a lot of work.

In honor of spooky month, share a 4 word horror story that only someone in your profession would understand. “Fifty-two bit mantissa.”

Datadog must have been polishing this piece of ragebait for years. It's art, really. (But seriously, this API is like a cheese grater to the face.)

Vibe-writing in Excel? No thanks 🤮 Unreliable tools can be incredibly useful - but you need a bulletproof test for their output. And spreadsheets are famously hard to get right! 94% of operational spreadsheets contain errors, and humans are exceptionally bad at catching them.

Best convo on the internet right now @vicvijayakumar.com

How AWS S3 serves 1 petabyte per second on top of slow HDDs // Stanislav Kozlovski Obscure feature + obscure feature + obscure feature = bug // Michael Gibson, @antithesis.com %CPU utilization is a lie // @brendanlong.bsky.social

This is *so* clever! At least in my head, this rhymes with the MapReduce philosophy of moving compute to the data.

I’d rather write statsd to the wire by hand than descend into the Lovecraftian horror of the OTel Go docs.

Unexpected benefit of saving PDFs of all the papers and @oreilly.bsky.social books I’ve read over the years: I have the most awesome NotebookLM session for distributed systems.

Find out why AI code gen delivers only partial wins today. Join us on Oct 9 for a live webinar with @antithesis.com's Will Wilson & @akshayshah.org. They'll share how to bridge the gap between AI generation and real engineering productivity. Don't miss it ➡️ bit.ly/Antithesis-A...

HBR is usually hot garbage, but you gotta respect whoever came up with “workslop” 💯 hbr.org/2025/09/ai-g...

“Antithesis’s deterministic simulation and active exploration remove the blindfold, enabling a systematic and reproducible search for bugs.” - Marek Siarkowicz, Kubernetes maintainer www.cncf.io/blog/2025/09...

Shmargaret Shmitchell, indeed.

Has anyone figured out how to get Claude to fix real bugs in distributed DBs? Hitting a brick wall with etcd and the long-abandoned Redis-Raft.

Excited to talk agents, AI, and enduring quality with Will and TNS.

So glad that testing/synctest is available without feature flags in Go 1.25! 🫡 To whichever Parse engineer originally wrote facebookgo/clock: you (and the many, many forks over the years) are finally off the hook. Thanks for paving the way.

Chroma has drawn some interesting conclusions applying property-based testing to LLMs: research.trychroma.com/generative-b... (h/t Hammad Bashir, who’s presenting this live at Chroma HQ!)

LLMs are remarkably effective at showing us - through our own automation choices! - which internal processes are just scar tissue that nobody cares about anymore.

Somehow I missed the news that EKS has also migrated off standard etcd. GKE is using Spanner. Is Azure still using OSS etcd? aws.amazon.com/blogs/contai...

I wonder what Roy Fielding's take on LLMs is - there's finally a way for computers to autonomously drive RESTful APIs.

Bemused by the enthusiastic embrace of JSON-RPC by the LLM tools ecosystem.

“The past few decades have seen almost unimaginable advances in compute performance and efficiency” My dude it’s only been 80 years since ENIAC. venturebeat.com/ai/why-the-a...

Finally accepting that I have to migrate my nvim config to Lua - I don't *want* to be vi-dependent, but I'm too far down this road to change now. The worst part is that the most critical nvim behavior is some cursed vimscript I wrote forever ago and barely understand now.

✍️ I wrote a new blogpost about two podcast episodes I find particularly interesting — spoiler: they’re from @oxide.computer 🖥️ and @antithesis.com 🧪 🔗 pierrezemb.fr/posts/debugg...

If you're into databases and you're in Seattle, @marcbrooker.bsky.social is talking at Seattle Systems on Aug 21: lu.ma/umtmqb38

EDB Engineering Newsletter #9 is now out! This time we're featuring posts on Postgres, distributed systems, benchmarking, AI, and performance engineering from Gunnar Morling, Antithesis/Jepsen, Justin Jaffray, Anthropic, Brendan Gregg, Simon Willison and more. edbeng.substack.com/p/edb-engine...

Recording for those who missed the talk: www.youtube.com/watch?v=Xdg3...

I had a great time talking to Ben and Will from Antithesis on the BugBash podcast about software testing and validation at AWS. In this wide-ranging hour we talk deterministic simulation testing, fuzzing, formal methods, the history and future of testing, and much more.

Ever wonder where deterministic simulation testing came from? On Episode 2 of the Bug Bash podcast, hear @marcbrooker.bsky.social talk to Will and Ben about how AWS achieves their legendary reliability. youtu.be/UGdMf7tC25Y open.spotify.com/show/5GtS7mZjc… podcasts.apple.com/us/podcast/the���

This release bundles an MCP server into gopls!

@brancz.com @asubiotto.com Are any Polar Signals engineers passing through San Francisco this fall? I'm looking for some speakers for a local meetup, and www.polarsignals.com/blog/posts/2... would make for a fantastic talk.

Today, FoundationDB is critical infra at Snowflake, Deepseek, Apple, and many other companies. But it started with some guys in a garage, building a demo with plywood and light bulbs... Hear the full story, told by the FDB founders, on the first episode of Antithesis's new BugBash podcast. #databs

Can’t get better than Antithesis and Jepsen together!

🚨 BSL incoming! 🚨 There’s no way that ClickHouse will keep letting competitors build on open source ClickHouse. clickhouse.com/blog/clickst...

Hatching Twitter was a great read. On to Character Limit! bookshop.org/p/books/hatc...

Can’t get over how much engineering effort went into Confluent’s Freight offering, only for it to be sold at a price point that’s still unfathomably enormous for high-volume workloads. The whole thing was just nuked from orbit by the innovator’s dilemma.

Huh - TIL that Motherduck isn’t actually the company behind DuckDB.