New blog post about Qbeast and how it brings a multidimensional spatial index to Iceberg/Delta. 🔹 Hypercube-based layout 🔹 Index used by writers, invisible to engines 🔹 Better locality + pruning, adaptive layout Lots of innovation ahead in lakehouses. jack-vanlightly.com/blog/2025/11...

Stream-order vs batch-order in Iceberg: * Flink wants temporal locality. * Spark wants value locality. Same table, conflicting physics. New post: jack-vanlightly.com/blog/2025/11...

Three KIPs (1150, 1176, 1183) all target Kafka’s cross-AZ replication costs but there is a wider question at stake. My new post explains the KIPs, the trade-offs between reusing old abstractions vs. embracing stateless compute over S3. jack-vanlightly.com/blog/2025/10...

New post: why I’m not a fan of “zero-copy” Iceberg tables for Apache Kafka. From a systems design view, it trades storage savings for coupling and complexity. Sometimes, duplication is cheaper than coupling. jack-vanlightly.com/blog/2025/10...

Why don’t Iceberg or Delta Lake have secondary indexes? Because analytics workloads and OLTP workloads optimize for opposite I/O patterns. See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/2025/10...

New deep dive: Understanding Apache Fluss I spent August reverse-engineering Fluss, Alibaba’s new table storage engine for Flink (partially forked from Kafka). This post covers its architecture, tiering, and how it tackles changelogs & low-latency state. jack-vanlightly.com/blog/2025/9/...

New blog post: A Conceptual Model for Storage Unification. The post defines what storage unification means, defines terminology and evaluates different building blocks and approaches to doing it. jack-vanlightly.com/blog/2025/8/...

In a future of autonomous AI agents, we can't limit ourselves to error prevention and error detection, we must also include remediation. jack-vanlightly.com/blog/2025/7/...

Science moves slowly because wrong theories waste decades. Engineering is careful because failures kill people. Software moves fast because mistakes are cheap, the expensive error isn't making the wrong choice, it's taking too long to make any choice. jack-vanlightly.com/blog/2025/7/...

Where does reliability begin, and where does it end? In distributed business architectures, the answer is responsibility boundaries. New post: jack-vanlightly.com/blog/2025/7/...

ChatGPT thought it was Tuesday, so I made fun of it and it admitted it was Wednesday. So I made fun of it again, and it admitted it was...Wednesday. But sure, AI agents are gonna steal my job 🤔

ChatGPT has hallucinated so many times for me today. It's invented scientific terms that don't exist, has been quite liberal with plausible answers based on what sounds reasonable, but without any real world justification. When challenged, it admits it's mistake.

My musical evolution continues, discovered deep hypnotic drone music today. No drugs required 😄 The Hypnus Records label is great.

How to reliably distribute work across microservices, stream processors, durable execution, event-driven, orchestration and now AI agents? Coordinated Progress is a 4 part series that explores the common structure behind reliable distributed systems. jack-vanlightly.com/blog/2025/6/...

I took a break from social media and my blog for a couple of months. ND burnout. But I'm tentatively back, probably just to post my writing here for now. HOTDS is on pause. Getting back to writing is therapeutic though. I'll post something this week that I've been working on.

Another Humans of the Data Sphere is out, with issue 10! In this issue people are talking fsyncs, tips for running ClickHouse at scale, the problems with MCP and more. Plus I dig up a classic paper from 1962. www.hotds.dev/p/humans-of-...

Proud to have contributed formal verification (TLA+) for three key improvements in Kafka 4.0: ✅ KIP-966: Strengthens the replication protocol. ✅ KIP-996: Introduces PreVote for more stable KRaft leadership. ✅ KIP-848: Delivers more efficient, predictable rebalancing.

Wow, I just discovered gamma wave music. Wrote non-stop for three hours.

Any Principal Engineers out there with ADHD or creative wiring — who don’t thrive in the tasks of project coordination, alignment meetings, and people management, but thrive on strategy, system design, writing, and shaping direction through ideas? Curious how you navigate the role.

A new disaggregated log replication survey post is out. How does the combination of Apache Pulsar with Apache BookKeeper divide and conquer the responsibilities of log replication? jack-vanlightly.com/blog/2025/3/...

I think I have an issue with tabs, it's grown to 371. My workstation is struggling to open chrome after a restart now.

Another Humans of the Data Sphere is out, with issue #9! In this issue, we also look at whether software engineers can learn from mechanical engineering, and looking at table formats as a form of virtualization. www.hotds.dev/p/humans-of-...

A new log replication disaggregation survey post is out! The Kafka Replication Protocol: 🔹Separation of control plane from data plane. 🔹Role separation with minimal coupling. 🔹Kafka’s alignment with Paxos roles. jack-vanlightly.com/blog/2025/2/...

Spotify is so bad at recommendations, but ChatGPT is pretty good at it. I give it a song and it lists the different characteristics of the song, then makes a set of recommendations based on those different characteristics.

The first post in the survey of disaggregated log replication systems is out! It looks at Neon's serverless Postgres write-path, which weaves consensus from heterogeneous components, based on MultiPaxos. jack-vanlightly.com/blog/2025/2/...

I updated my How to Disaggregate a Log Replication Protocol to include "separating ordering from IO". Basically I couldn't ignore CORFU as a way of separating responsibilities! So now we have A-F of ways of breaking apart the monolith. jack-vanlightly.com/blog/2025/2/...

Table virtualization, stream-table materialization redefine how we think about data sharing, composability, and interoperability. The OTFs (Iceberg, Delta Lake), metadata separation, & cloud storage are paving the way for modular, composable data platforms. jack-vanlightly.com/blog/2025/2/...

So many databases claim to be compatible with #Postgres these days, but what does that really mean? @pgdba.bsky.social tries to provide an objective answer with the Postgres Compatibility Index, which verifies compatibility in a wide range of criteria. Loving to see this effort!

The latest Humans of the Data Sphere is out, with issue #8! Additional topics in this post include systems correctness practices at AWS, Datadog's Husky compaction and solving for the distributed case first in ambitious systems projects. www.hotds.dev/p/humans-of-...

This reminds me of Paxos vs Raft. Paxos formalized the responsibilities of reaching consensus and acting on the agreed values into *distinct roles* (proposer, acceptor, learner). These roles can be put in a monolith or distributed. The basis for the Paxos family of protocols are these roles.

Confluent + Databricks next-level partnership 💪 Bi-directional flow between Confluent and Databricks. Kafka topics appearing as Delta tables in Databricks. Delta tables appearing as Kafka topics in Confluent. Simply amazing.

New dist sys post on log replication! In this one, I classify five approaches to disaggregating log replication protocols—laying the groundwork for a survey of real-world systems through the lens of disaggregation. jack-vanlightly.com/blog/2025/2/...

Yesterday, I covered Virtual Consensus—today, let’s dive deeper into a crucial distinction: 👉 Failure-free ordering vs. Fault-tolerant consensus Decoupling these concepts can change how we think about consensus protocols. jack-vanlightly.com/blog/2025/2/...

New distributed systems protocol write-up! This write-up dives into the Virtual Consensus in Delos paper and why it makes sense as the default log replication protocol in the era of object storage and hybrid environments. jack-vanlightly.com/blog/2025/2/...

Speculation is growing that Snowflake is planning to acquire Redpanda—but why? What justifies the rumored high price tag? When you consider market trends, the Snowflake vs. Databricks rivalry, and the AI shift, the rationale starts to become clear. Here’s my take. jack-vanlightly.com/blog/2025/2/...

Humans of the Data Sphere issue #7 is out. Obviously, DeepSeek happened but there have been plenty of other conversations happening. In this issue we also look at well and ill-conditioned APIs and the challenge of contextual data quality in AI-powered data pipelines. www.hotds.dev/p/humans-of-...

I read this blogpost 3 times, not interested in the investment aspect, but there's so many valuable insights in here on current development of AI. Also the writing alone deserves its own separate praise. youtubetranscriptoptimizer.com/blog/05_the_...

I had RSI in my wrists so bad 20 years ago (too much Counterstrike) that I resorted to typing using pencils with rubbers on the end. I would basically jab the keys and mouse buttons, holding the pencils in my fists. It looked ridiculous but it allowed me to carry on working.

AI-related stocks are selling off because DeepSeek’s new model was cheaper to train and has lower inference costs. I suppose investors see that and see a threat to the AI industry. But they’ve got it backwards...

Regarding Restate and its distributed log, many people talk about Delos but Apache BookKeeper is also highly relevant/similar, so I like to remind people that it also exists! I've written extensively about how BookKeeper works if you're interested: 1\ medium.com/splunk-maas/...

Great stuff. I'm watching the durable execution space closely and personally I'm quite bullish on it. I'll be writing my own thoughts on durable execution soon.

Over the last week I've been working on finishing the formal verification of Kafka's KRaft with pre-vote and reconfiguration. The protocol is looking good from a design perspective. The team use deterministic simulation testing to catch implementation bugs. Defensive in depth!

The pace of AI advancement is not slowing down, and now AI agents are breaking onto the scene, leveraging ever more powerful models to interact with the real world. In this post, I cover some of the voices talking about AI agents and the challenges ahead. jack-vanlightly.com/blog/2025/1/...

My top favorite quotes in issue #6 of HOTDS are: 1) Marc Brooker's blog post: Snapshot Isolation vs Serializability.

Another issue of Humans of the Data Sphere is out, with issue #6! In this issue we also dive into the world of AI agents including the promises and challenges ahead. www.hotds.dev/p/humans-of-...

I've not seen much talk of the tragedy of the commons and the potential wave of AGI in the future. The common resource is consumer demand. Companies may replace their humans with AGI for short term gain, not considering the collective long-term impact on consumer demand and therefore the economy.

I've starting compiling the next issue of Humans of the Data Sphere 🙌, while monitoring the model checker chugging its way through the Kafka KIP-996 (prevote) KRaft TLA+ spec. If the issue isn't out tomorrow then I probably got drawn back into TLA+ land.

I'm back from vacation, putting down Splatoon 3 and picking up work projects again. Kafka transactions verification work will resume next week and I'll carry on with the diary.

It's the holidays, but I still carve out a little time of Kafka transactions. Diary entry 5 - adding AddPartitionsToTxn request/response, TC elections and producer fencing (TLA+). jack-vanlightly.com/...

I now have FindCoordinator, InitPidRequest, AddPartitionsToTxnRequest implemented. Also with coordinator elections moving the coordinator around. Next up is a client writing records to partitions. Maybe 1/3 of the way.