The latest Jepsen talk, from Systems Distributed in June, goes live in 15 minutes. We'll be doing a live chat during the premier, if you want to chat about databases and testing. :-) https://www.youtube.com/watch?v=dpTxWePmW5Y

A new #Jepsen report! We tested early builds of Capela, an unreleased distributed programming environment, and found twenty-two issues, including four language problems, fourteen crashes or non-fatal panics, performance degradation, and three safety issues including lost update […]

An interview with Kaivalya Apte, on The GeekNarrator podcast. We talk about mapping properties to tests, type I and II errors, performance, LLMs, and more. https://www.youtube.com/watch?v=IvE1VbOol88

The video of my BugBash talk, "Jepsen 17: ACID Jazz" is out now! https://www.youtube.com/watch?v=v8cG2hh10SQ

Antithesis and Jepsen are releasing a glossary of terms useful in distributed systems testing: https://antithesis.com/resources/reliability_glossary/

Systems Distributed. June 19-20, Amsterdam. https://systemsdistributed.com/

A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]

Jepsen 0.3.9 is now available, including a module for restarting flaky databases, a few improvements to downloading logs, more capable generators, and friendlier error messages. https://github.com/jepsen-io/jepsen/releases/tag/v0.3.9

A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]

Added four new phenomena to Jepsen's docs: P4 (Lost Update), A5A (Read Skew), A5B (Write Skew) and Process. https://jepsen.io/consistency/phenomena#sql

Jepsen 0.3.8 is now available. It includes a new nemesis for file corruption, and improvements to clock-skew tests. https://github.com/jepsen-io/jepsen/releases/tag/v0.3.8

What IS Strong Serializability, really? Ever want to try writing your own gossip service. Two open sessions of Jepsen's training classes are coming up: the Distributed Systems Fundamentals class, and (for the first time!) its accompanying workshop […]

I'll be speaking on Jepsen at Bug Bash (DC, April 3-4), and Systems Distributed (Amsterdam, June 19-20). Come join! https://bugbash.antithesis.com/ https://systemsdistributed.com/

So, uh, the last time I had to call malloc() was a quarter century ago, and I am struggling to do basic tasks without corrupting the heap. I would love to hire one of you stripey-socked C witches for a very small contract to help finish […]

Added descriptions of the SQL isolation level anomalies P0, P1, P2, and P3 to the phenomena page: https://jepsen.io/consistency/phenomena#sql

Released version 0.2.4 of Maelstrom, Jepsen's workbench for writing toy distributed systems: https://github.com/jepsen-io/maelstrom

There's a few tickets left for the distributed systems class coming up in just over a week. If you'd like to join, now's the time. :-) https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon

Bin Wang put together a Jepsen test for Patroni, a PostgreSQL replication system. All sorts of good stuff in here, including that the cluster can't handle a series of single-node failures: https://www.binwang.me/2024-12-02-PostgreSQL-High-Availability-Solutions-Part-1.html

Antithesis, Buf, and Jepsen are running a joint webinar on December 5th. We'll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause […]

"That's never been my understanding of auto-commit, that would be a crazy default wouldn't it?" I feel like there's a real need for database user research--building a quantitative and qualitative picture of how actual users understand the systems they use […]

A new #Jepsen report! We worked with Buf to analyze the safety of Bufstream, a Kafka-compatible streaming system. We found three safety and two liveness issues in Bufstream 0.1.0, including the loss of acknowledged writes in healthy clusters. These problems were resolved by version 0.1.3 […]

Last year we reported that MySQL and MariaDB's "Repeatable Read" was badly broken. The MariaDB team has been hard at work, and they've added a new flag, `--innodb-snapshot-isolation=true`, which causes "Repeatable Read" to prevent Lost Update, Non-repeatable Read, and violations of Monotonic […]

If you've been waiting for a chance to take Jepsen's distributed systems class, we've got great news. We're running a special session December 16-19 that anyone can register for. https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon

Just want to give a shout out to Irene Kannyo, who's edited every Jepsen analysis over the last four years. She brings her background as scientist and writer to our industry-focused reports, maintains an evolving style guide, streamlines prose, ensures diagrams match descriptions, and so much […]

#Jepsen 0.3.6 is now available on GitHub and Clojars. This is a sizeable release. It includes a significant correctness bugfix for a rare bug that could make operations in the history print with the wrong data. It also adds a new namespace for composing databases, nemeses, and generators when […]

Just a heads-up: I'll be speaking at GOTO Chicago next Tuesday, 2024-10-24, on JVM performance techniques in Jepsen: https://gotochgo.com/2024/schedule?date=2024-10-22&video=false

Jepsen 16: I Can't Believe It's Not ACID, delivered at Systems Distributed 2024. RavenDB, MariaDB/MySQL, RedPanda. https://www.youtube.com/watch?v=ecZp6cWhDjg#t=45s

A new, short #Jepsen report is out on jetcd 0.8.2. Jetcd is the official Java client library for etcd. We show that it contains an improper retry mechanism which causes lost update, circular information flow, and aborted read. I hope you enjoy it! https://jepsen.io/analyses/jetcd-0.8.2

I'll be speaking at GOTO Chicago on performance techniques in Jepsen. It's gonna be a weird collage of parallelism, pure functions, immutable data structures, and deforestation with low-level techniques like bitsets, avoiding sharing between threads, packing structures into mutable arrays […]

Jepsen has a blog now, with RSS and Atom feeds: https://jepsen.io/blog.

I'm going to speak about MySQL, RavenDB, and Datomic at Systems Distributed 2024, June 27 & 28th, in New York: https://systemsdistributed.com/#about-conference

Pleased to announce a new #Jepsen report on Datomic Pro 1.0.7075. Datomic is an unusual system in many ways. Temporal! Immutable! Datalog! Strong Session Serializable*! We found that Datomic appears to offer Strong Session Serializable inter-transaction semantics, up to Strong Serializable when…