Great recent find “Epic: The Musical”., based on The Odyssey. Gods, monsters, magic, love, death, betrayal, revenge, drama. It has It all. **So quotable**. music.youtube.com/playlist?lis...

#SD25 online continues today! Join Kyle Kingsbury for Jepsen 18: Serializable Mom How three unconventional systems ensure--or violate--key safety properties 10am PT / 1pm ET / 7pm CET www.youtube.com/watch?v=dpTx... @jepsen.mastodon.jepsen.io.ap.brid.gy

------- "Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25 Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases. "I professionally set those tires on fire".

It's day two of Systems Distributed, hosted by @tigerbeetle.com! I'll be liveskeeting all of the talks, except mine (at 11 AM). Since the venue is a film museum, they're setting up special posters for each talk. #sd25

Greetings from #sd25! It's a two-day single-track conference hosted by @tigerbeetle.com. I'll be aiming to liveskeet as many of the talks as I can. Super excited to be here! systemsdistributed.com

A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]

A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]

All the videos are up at www.hytradboi.com/2025#program now.

"Correctness at Feldera" talks about various correctness techniques, including: - machine proof of the underlying DBSP algorithm - differential testing of the implementation https://www.feldera.com/blog/correctness-at-feldera

Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...

Curated list of materials on testing SQL database engines is public now github.com/asatarin/tes...

This is a sample from a list "testing-sql-databases" I have. It's more drafty, most likely missing a ton from big tech, startups and academia alike and not published.

Good old Microsoft published some work: - "Deploying a Steered Query Optimizer in Production at Microsoft" dl.acm.org/doi/abs/10.1... - This great talk "The Cascades Framework for Query Optimization at Microsoft" touches on correctness youtu.be/pQe1LQJiXN0

Work on correctness of optimizers from Greenplum - "Automatic capture of minimal, portable, and executable bug repros using AMPERe" dl.acm.org/doi/10.1145/... - "Testing the accuracy of query optimizers" dl.acm.org/doi/10.1145/...

Similar work from Databricks: - "SparkFuzz: searching correctness regressions in modern query engines" dl.acm.org/doi/abs/10.1... - "Correctness and Performance of Apache Spark SQL" youtu.be/fddBOZxdUKI

You already mentioned "Snowtrail: Testing with Production Queries on a Cloud Database" from Snowflake dl.acm.org/doi/10.1145/...

More correctness work from MongoDB, not yet on my list "Verifying Transactional Consistency of MongoDB" arxiv.org/abs/2111.14946

MongoDB team did great work on performance: - "Creating a Virtuous Cycle in Performance Testing at MongoDB" dl.acm.org/doi/10.1145/... - "The Use of Change Point Detection ..." dl.acm.org/doi/abs/10.1... - "Fair Benchmarking Considered Difficult" mytherin.github.io/papers/2018-...

To counter balance your argument some big tech work on correctness of (SQL) database in a thread below. Spanner has incredibly sophisticated random generated checks internally, this just scratches the surface: - "Randomized Testing of Cloud Spanner" medium.com/@jcorbett_26...

Get your point, but to be fair, this list is not targeting correctness of databases (SQL or otherwise). It almost entirely excludes anything single node or targeting single threaded execution (like a fuzzer), with some late additions in asatarin.github.io/testing-dist...

I'd be happy to learn that this is just a gap in my testing knowledge, or that there's a bunch of secret testing systems in big companies that I don't know about, but even just skimming the list of asatarin.github.io/testing-dist... kind of illustrates my point. (Thanks again @asatarin.bsky.social!)

Deterministic simulation has been entirely pushed by startups. Jepsen is jepsen. All the SQL fuzzing stuff I know of comes from academia or database startups. Larger companies seem to only be winning in the application of formal methods (because they can afford to hire a team of ex-professors?)

When I think of advancements in quality and correctness of databases, it feels most things have come from startups or individuals, and not large well-established companies. Which seems... backwards? We talk of startups hacking out code and megacorps crawling to keep a high quality bar.

Alexey's @clickhouse.com talk here is hilariously off-the-chain. I highly recommend watching it: www.youtube.com/watch?v=jmVx... You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.

MariaDB is improving their isolation level guarantees following the recent @jepsen.mastodon.jepsen.io.ap.brid.gy report. "Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.

Alexey Gotsman talks about database isolation levels. Starts off with consistency in general: - what is strong consistency - common misunderstandings and pitfalls - serializability and linearizability - weak isolation levels - safety and liveness Video links below.

Hi, I’m a distributed systems researcher at MongoDB, photographer, Zen Buddhist in the White Plum lineage, rock climber living in New Paltz NY. I hope to meet fellow researchers here, and I hope we can all settle on BlueSky as our Schelling point for a couple years before we have to do this again.

Hello, Bluesky users! I curate and maintain list of resources on testing distributed systems. You might have seen it before. It's a good one, if I may say so myself. asatarin.github.io/testing-dist...