How DataDog built a low-latency, multi-tenant data replication platform with Kafka, Kafka Connect, Debezium, and Temporal www.datadoghq.com/blog/enginee...

Having multiple copies of your data, specifically laid out for different access patterns, is completely fine. Desirable even. Just make sure to have one canonical source of truth and drive updates to all the copies from there.

Stream-order vs batch-order in Iceberg: * Flink wants temporal locality. * Spark wants value locality. Same table, conflicting physics. New post: jack-vanlightly.com/blog/2025/11...

Looks like that post is not going down too well over on the 🍊 site, LOL. Genuinely curious why someone would flag it, but I guess we'll never know.

💯 we had an entire "logs" database at WePay and it was a nightmare. It's convenient until it isn't. And backing out is brutal.

📝 Blogged: "'You Don't Need Kafka, Just Use Postgres' Considered Harmful" In which I'm arguing that both Postgres and Kafka are great tools for their respective purposes. But don't create your custom implementation of one on top of the other. 👉 www.morling.dev/blog/you-don...

With nary 8.5 hours to spare (GMT) before the end of the month, herewith a whole lotta links about things in the data engineering world that I found interesting this month. 👉 rmoff.net/2025/10/31/i...

I'm very excited about the upcoming webinar: "The State of Stream Processing", recorded with brilliant @olenakutsenko.bsky.social and @gunnarmorling.dev.

Ok, ok, enough already, working on that "'You Don’t Need Kafka, Just Use Postgres' Considered Harmful" post right now 🤣.

Learn how to import JFR data into DuckDB, to easily query and analyze your profiling data in this week's blog post: mostlynerdless.de/blog/2025/10...

Have you ever found yourself thinking…Flink Watermarks…WTF? 🔥 This new tool might be for you :) It's a hands-on, scrollytelling walkthrough of what watermarks in #ApacheFlink are, why they matter, and how to use them. Try it out: flink-watermarks.wtf #dataBS

Friends don't let friends deploy to us-east-1.

New post: why I’m not a fan of “zero-copy” Iceberg tables for Apache Kafka. From a systems design view, it trades storage savings for coupling and complexity. Sometimes, duplication is cheaper than coupling. jack-vanlightly.com/blog/2025/10...

"Beyond Indexes: How Open Table Formats Optimize Query Performance" Enjoyed reading this post by @vanlightly.bsky.social, explaining why secondary indexes, as known from OLTP systems, don't make sense for data lakehouse systems like Apache Iceberg. 👉 jack-vanlightly.com/blog/2025/10...

Why don’t Iceberg or Delta Lake have secondary indexes? Because analytics workloads and OLTP workloads optimize for opposite I/O patterns. See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/2025/10...

🏃 Tired but happy after finishing my longest race (12.3 km) so far, the Köhlbrandbrückenlauf. One of the few occasions throughout the year where you can cross the bridge over Hamburg's port without sitting in a car. What an amazing event!

The Kroxylicious project is very pleased to announce the release of v0.16.0 bit.ly/3IpOflV. Main changes bit.ly/4o3Eby5 are Kafka 4.1 compatibility and observabilty improvements. Happy proxying🐊.

Our SIGMOD paper with our friends at Tsinghua + @wesmckinney.com + @pateljm.bsky.social on creating a next generation open-source data file format is out. F3 is a future-proof file format avoids the mistakes of Parquet. 📄 Paper: db.cs.cmu.edu/papers/2025/... 📁 Code: github.com/future-file-...

It's DONE. SHIP IT. Finished my 2nd edition of Building Event-Driven Microservices, published by @oreilly.bsky.social and sponsored by the good people at @confluent.io. You can find this book on the O'Reilly learning platform (www.oreilly.com/library/view... - you get a free trial!),

Great day out at the 10K Alstertallauf today. As always, started with a higher pace than planned. But then, kinda to my own surprise, I could actually keep it up. Not too fast in absolute terms, but faster than yesterday's me 😊.

Free isn’t free: the infrastructure behind open source has real costs, and it’s time we aligned usage with responsibility. This morning we jointly launch a new blog and open letter on sustainable stewardship. www.sonatype.com/blog/from-ab...

Java 25 / JDK 25: Features since the previous long-term support (LTS) release: openjdk.org/projects/jdk... #Java25 #JDK25 #OpenJDK #Java

SIMD Within a Register: How I Doubled Hash Table Lookup Performance Learned about SWAR the first time in the one-billion row challenge. Nice explanation including benchmarks.

Has anyone tested the Iceberg write support new in DuckDB 1.4.0 yet? I'm using Polaris as a catalog; DuckDB uploads the actual data to my bucket (which is in MinIO), but then tries to commit the metadata on Amazon S3 🤔.

📝 Blogged: "Let's Take a Look at... Lower Java Tail Latencies With ZGC" As #Java 25 got released earlier this week, I took the ZGC garbage collector for a spin, exploring how it lives up to its promise of sub-millisecond GC pause times. 👉 www.morling.dev/blog/lower-j...

Today is the day, Java 25 went GA!

Java 25 / JDK 25: General Availability: mail.openjdk.org/pipermail/jd... Features: openjdk.org/projects/jdk... Downloads: jdk.java.net/25/ #Java25 #JDK25 #OpenJDK #Java

One of the great things about RDBMS: They do event sourcing, so you don't have to.

Let's Take a Look at... JEP 483: Ahead-of-Time Class Loading & Linking! buff.ly/vXDtFq9 #Java #jdk #jep483

"Peeking Inside the SQL Server Transaction Log" Enjoyed reading this post by Vojtech Juranek from the @debezium.bsky.social team, exploring the SQL Server TX log format. A potential alternative to polling SQL Server's CDC tables on the horizon? 👉 lnkd.in/eyKFXqZw debezium.io/blog/2025/09...

Don't be like Kenny--Java 25 is around the corner!

Don't be like Kenny--Java 25 is around the corner!

Seen this series (in progress) @gunnarmorling.dev ? youtu.be/n-YK3B4_xPA?...

Published a new blog post: Replacing a Cache Service with a Database We already use databases, why can't we use them to replace caches as well? Will we ever replace caches entirely with databases? In this post, I will share some ideas, & discuss how we are moving toward this avi.im/blag/2025/db...

Now up on stage at @javazone.bsky.social: @josepaumard.bsky.social on Loom, Amber and other goodness in recent Java versions. 21 and 25 named by him "current and next LTS"--take that, @nipafx.dev ;) To be fair, that Non LTS slide followed. I'm confused 😇.

Arriving on-site for @javazone.bsky.social ! So glad to be back for another year, this time with a lightning talk on the pitfalls around #Postgres replication slots (tomorrow, 9:40 am) and a breakout session about the the outbox pattern (Thursday, 10:20 am). 👉 2025.javazone.no/en/program

New deep dive: Understanding Apache Fluss I spent August reverse-engineering Fluss, Alibaba’s new table storage engine for Flink (partially forked from Kafka). This post covers its architecture, tiering, and how it tackles changelogs & low-latency state. jack-vanlightly.com/blog/2025/9/...

🌴 Back to work after some amazing summer holidays. What have I missed? Any cool data streaming news over the last few weeks I should catch up on?

Join @royvanrijn.com at #YOW25, where he shows how Java can fly in his talk "Pushing Java to the Limits: Processing a Billion Rows in Under 2 Seconds" MEL 4-5 Dec | BNE 8-9 Dec | SYD 11-12 Dec yowcon.com

I keep getting into conversations with engineers where I talk about how desperately I want my database to support materialized views with incremental updates and too often I get a blank stare in response. so I wrote a new post about why you should care! sophiebits.com/2025/08/22/m...

✍️ Blogged: Kafka to Iceberg - Exploring the Options rmoff.net/2025/08/18/k... #dataBS

Kroxylicious 0.15.0 released This is a fast-tracked release to address a key resilience issue: - Improved Bootstrap Resilience: The proxy now defaults to round-robin between brokers in bootstrapServers. - New record encryption 🔒quickstart Check out the full details kroxylicious.io/blog/kroxyli...

Whenever someone shares a performance comparison of programming languages, many folks are surprised to see #Java rank up very high. I think by now it's kinda criminal how underrated Java and the JVM are in circles not familiar with it. This misconception is the biggest challenge for this platform.

High-Performance Autocomplete Search with Quarkus, Panache, and PostgreSQL Power intelligent, real-time search experiences with fast indexing, efficient persistence, and a lean Java backend. buff.ly/BumYADc #Java #Quarkus #Hibernate #PostgreSQL #Autocomplete #pg_trgm

MySQL's New Storage and Execution Engine: DuckDB​ www.linkedin.com/pulse/mysqls...

EDB Engineering Newsletter #9 is now out! This time we're featuring posts on Postgres, distributed systems, benchmarking, AI, and performance engineering from Gunnar Morling, Antithesis/Jepsen, Justin Jaffray, Anthropic, Brendan Gregg, Simon Willison and more. edbeng.substack.com/p/edb-engine...

📣 Kroxylicious 0.14.0 - new AuthN APIs (more to come here) - new metrics for tracking network stalls - Fixing support for encrypting messages over 1mb (we don't judge the messages we proxy...) - Better support for negotiating protocol versions kroxylicious.io/blog/kroxyli...

Quite bummed to miss @devoxx.com this year due to a scheduling conflict. Seeing all the acceptance notes definitely creates FOMO 😵‍💫. Wishing everyone a great event! Hope to be back next year.

Maybe it's me being dense, but wouldn't Rust's `await` be more aptly named `yield`? With my Java hat on, await makes it sound like this is blocking (like Future::get()), whereas it's exactly not that.

IMO we need three major axes for talking about open software: - Open source: anyone can use, modify, redistribute the source code - Open contribution: anyone can help make it better - Open governance: anyone can wield power over what happens to it