Oh, that's a useful feature. You can enforce causal consistency in general (which includes read-your-writes and other guarantees) with that mechanism. We've had that in MongoDB for some time, though, and customers have found it hard to use: emptysqua.re/blog/how-to-...

Happy Holidays, and to those who celebrate: Merry Christmas 🎄🎁☃️! Enjoy the days ahead ("between the years", as we call it in German), and of course: yippee ki-yay!

🐘 Oh, that's nice: Postgres 19 is going to ship read-your-writes for standbys when using asynchronous replication, via a new command WAIT FOR LSN (similar to WAIT_FOR_EXECUTED_GTID_SET in MySQL). Very cool! www.postgresql.org/docs/devel/w...

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet. Presumably, the design of yore is not going to cut it moving forward. I spent some time to understand a bit better how things actually changed. sympathetic.ink/2025/12/11/C...

Why write engineering blogs? Here’s how some of our favorite bloggers responded to the question “Why did you start blogging and why do you continue?” writethatblog.substack.com/p/why-write-...

📝 Blogged: "You Gotta Push If You Wanna Pull" Wrote up some thoughts on push vs. pull queries, materialized views, and why data duplication isn't something to fear. 👉 www.morling.dev/blog/you-got...

A key factor for making technical discussions more successful (and enjoyable) is to resist the urge to one up each other. Don't try to prove you're smarter than others, instead work collaboratively towards a solution, benefiting everyone involved.

"Streams for machines, tables for humans". Kinda proud of this tag line from my upcoming blog post, discussing how pull and push queries go hand in hand in data management systems. Out early next week 🤓.

New post: The Durable Function Tree. Durable execution engines all end up building some form of function tree with suspension points shaped by local vs remote side effects. I look at why, the trade-offs, and where orchestration should (and shouldn’t) be used. jack-vanlightly.com/blog/2025/12...

"Mini-LSM--Build a simple key-value storage engine in a week" This looks like a great learning resource for folks looking to get started with log-structured merge trees in #Rust 🦀, by Alex Chi Z. 👉 skyzh.github.io/mini-lsm/

PG DATA CfP is open! We’re building a new Postgres conference series focused on education, community, and collaboration. 👉 Submit your talk proposal here: sessionize.com/pg-data-2026 Join us in June! #PGData2026 #PostgreSQL #PostgresCommunity #MidwestTech #ChicagoTech

📺 "Ins and Outs of the Outbox Pattern" An invaluable pattern to ensure a service can update its own database and send messages to other services via Kafka, consistently. In this talk I'm discussing implementation options, backfills, alternatives, etc. www.youtube.com/watch?v=Pkrz...

Wait what's this?! I've managed to pull my finger out to ship a list of **a whole lot of interesting links about stuff going on in the world of data for November**, a whole three days left until the end of the month! rmoff.net/2025/11/26/i...

🗣️ "The combination of AVRO’s binary encoding and lz4 compression delivered exactly what we wanted: an order-of-magnitude smaller footprint, faster network transfers and reduced latency" Great post by Emiliano Mancuso about the benefits of Avro for CDC pipelines 👍. 👉 medium.com/fresha-data-...

Building a Durable Execution Engine With SQLite Great article with accompanying Java code, that uses annotations, bytebuddy and sqlite to be able to resume jobs. Great to follow explanations plus code.

Java 25's new CPU-Time Profiler (1) buff.ly/70nU5zN #Java

📝 Blogged: "On Idempotency Keys" Discussing several options for ensuring exactly-once processing in distributed systems using idempotency keys, from UUIDs to monotonically increasing sequences. 👉 www.morling.dev/blog/on-idem...

👇Excellent post by @vanlightly.bsky.social, discussing why control flow code needs to be deterministic in Durable Execution. Great read!

Livelock exists in all forms of leaderless Paxos, and you need to either find a way to negotiate your way out of it, or accept that the things can commute and generalized Paxos your way into tolerating it. Rystsov also proposed a cute trick around this in web.archive.org/web/20231207...

Last week I started exploring in C# the (originally Java) One Billion Row Challenge (github.com/gunnarmorlin...). This is a really interesting conversation with the creator, @gunnarmorling.dev: www.youtube.com/watch?v=RYjB...

Spent some time digging into the #CASPaxos paper ("Replicated State Machines without logs"), by Denis Rystsov. It's really easy to follow; do I get it right though that two competing writers could block each other from ever moving beyond the prepare phase? arxiv.org/pdf/1802.07000

📺 "Distributed Systems 6.2: Raft" Enjoyed this intro to the Raft consensus algorithm, part of a larger DistSys lecture series by @martin.kleppmann.com. The pseudo-code makes it actually digestible really well. Great watch! www.youtube.com/watch?v=uXEY...

Now on the 🍊 site. Dare I to read the comments 🤣?

👇ICYMI: In my new blog post I'm diving into the fundamentals of Durable Execution, which makes multi-step workflows, defined in code, persistent and resumable. Featuring #SQLite as an execution log as well as some of the goodies of modern #Java, such as virtual threads and scoped variables.

Getting an AI-code-is-technical-debt vibe sometimes... I love accruing it. When the bill comes due and I need to dig in and figure out why things aren't working.... :grimacing:

📝 Blogged: "Building a Durable Execution Engine With SQLite" Exploring the fundamentals of DE, based on a minimal engine built from scratch, using #SQLite as an execution log, and modern #Java (👋 virtual threads). 👉https://www.morling.dev/blog/building-durable-execution-engine-with-sqlite/

A deep dive into Apache Kafka's KRaft protocol developers.redhat.com/articles/202... - Good article from Federico Valeri

"Help! My LocalDate Isn’t Flattened" Enjoyed reading this post by @horstmann.com about the subtleties of turning Java's LocalDate into a value type for Project Valhalla. Null-ability and built-in object serialization keep haunting us. horstmann.com/unblog/2025-...

Hi everyone! An ongoing reorganization at my current employer has prompted me to seek for a new role and would appreciate your support. I'm open to remote (anywhere) and hybrid roles in Switzerland. I have 25+ years experience as a software developer, 8 as devrel www.linkedin.com/posts/aalmir...

Three Flink-related talks at P99 CONF this year - here are the recordings and slides: * www.p99conf.io/session/apac... * www.p99conf.io/session/buil... * www.p99conf.io/session/rivi...

Call for Papers for Current26 in Bengaluru and London is now open! Share your talks on Data Streaming, Kafka, or the future of AI in production etc with the world's most innovative data community. Deadline December 22nd 2025 Bengaluru ➡️ https://cnfl.io/49L8V2J London ➡️ https://cnfl.io/47IBj2Y

How DataDog built a low-latency, multi-tenant data replication platform with Kafka, Kafka Connect, Debezium, and Temporal www.datadoghq.com/blog/enginee...

Having multiple copies of your data, specifically laid out for different access patterns, is completely fine. Desirable even. Just make sure to have one canonical source of truth and drive updates to all the copies from there.

Stream-order vs batch-order in Iceberg: * Flink wants temporal locality. * Spark wants value locality. Same table, conflicting physics. New post: jack-vanlightly.com/blog/2025/11...

Looks like that post is not going down too well over on the 🍊 site, LOL. Genuinely curious why someone would flag it, but I guess we'll never know.

💯 we had an entire "logs" database at WePay and it was a nightmare. It's convenient until it isn't. And backing out is brutal.

📝 Blogged: "'You Don't Need Kafka, Just Use Postgres' Considered Harmful" In which I'm arguing that both Postgres and Kafka are great tools for their respective purposes. But don't create your custom implementation of one on top of the other. 👉 www.morling.dev/blog/you-don...

With nary 8.5 hours to spare (GMT) before the end of the month, herewith a whole lotta links about things in the data engineering world that I found interesting this month. 👉 rmoff.net/2025/10/31/i...

I'm very excited about the upcoming webinar: "The State of Stream Processing", recorded with brilliant @olenakutsenko.bsky.social and @gunnarmorling.dev.

Ok, ok, enough already, working on that "'You Don’t Need Kafka, Just Use Postgres' Considered Harmful" post right now 🤣.

Learn how to import JFR data into DuckDB, to easily query and analyze your profiling data in this week's blog post: mostlynerdless.de/blog/2025/10...

Have you ever found yourself thinking…Flink Watermarks…WTF? 🔥 This new tool might be for you :) It's a hands-on, scrollytelling walkthrough of what watermarks in #ApacheFlink are, why they matter, and how to use them. Try it out: flink-watermarks.wtf #dataBS

Friends don't let friends deploy to us-east-1.

New post: why I’m not a fan of “zero-copy” Iceberg tables for Apache Kafka. From a systems design view, it trades storage savings for coupling and complexity. Sometimes, duplication is cheaper than coupling. jack-vanlightly.com/blog/2025/10...

"Beyond Indexes: How Open Table Formats Optimize Query Performance" Enjoyed reading this post by @vanlightly.bsky.social, explaining why secondary indexes, as known from OLTP systems, don't make sense for data lakehouse systems like Apache Iceberg. 👉 jack-vanlightly.com/blog/2025/10...

Why don’t Iceberg or Delta Lake have secondary indexes? Because analytics workloads and OLTP workloads optimize for opposite I/O patterns. See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/2025/10...

🏃 Tired but happy after finishing my longest race (12.3 km) so far, the Köhlbrandbrückenlauf. One of the few occasions throughout the year where you can cross the bridge over Hamburg's port without sitting in a car. What an amazing event!

The Kroxylicious project is very pleased to announce the release of v0.16.0 bit.ly/3IpOflV. Main changes bit.ly/4o3Eby5 are Kafka 4.1 compatibility and observabilty improvements. Happy proxying🐊.

Our SIGMOD paper with our friends at Tsinghua + @wesmckinney.com + @pateljm.bsky.social on creating a next generation open-source data file format is out. F3 is a future-proof file format avoids the mistakes of Parquet. 📄 Paper: db.cs.cmu.edu/papers/2025/... 📁 Code: github.com/future-file-...

It's DONE. SHIP IT. Finished my 2nd edition of Building Event-Driven Microservices, published by @oreilly.bsky.social and sponsored by the good people at @confluent.io. You can find this book on the O'Reilly learning platform (www.oreilly.com/library/view... - you get a free trial!),