This was posted 15 years ago (part 1/2)

**taps the sign**

when I say “storage is cheaper now” this is what I mean topicpartition.io/definitions/...

“KIP-1150 introduces Diskless Kafka topics that write directly to S3 instead of replicating between brokers.” “Even using the expensive S3 Express (which a week ago lowered its prices by more than 50%) still saves 73% compared to traditional Apache Kafka.” /ht @ananthdurai.bsky.social

a new 2 minute streaming post is sitting patiently in your inbox... open it to learn when: • Kafka decides what messages are visible to Consumers • acks=all Producers receive responses

Apache Kafka has been on a Diskless craze in the last two years: • 2023: WarpStream launched • 2024: Confluent bought them for $220M+ • 2025: Aiven published a KIP to the open source project to introduce the same type of leaderless, direct-to-S3 topics

Yesterday, CloudFlare dropped a bomb that I believe may change the future of Lakehouse storage. R2 + Iceberg should become the de-facto choice for hybrid and multi-cloud data lakehouse architectures. Here's why it may break the cloud monopoly 🧵

+1 to this future

This is the most impactful Apache Kafka mentor you’ve never heard of: Chia-Ping Tsai. In just 18 months, he bootstrapped a large Taiwanese open source community boasting: • 5000 participants • 15,000 Slack messages/month • 10 meetings/week • 20+ Apache committers

why doesn't Kafka use Protobuf or another popular serialization format? why is everything custom? afaict the decision to go custom was taken back when it was first created. I just assume we never questioned it again? ever since it's been a one way street - since upgrading clients will be a pain

What if I told you that a 1 GiB/s Kafka topic streamed directly into your S3 data lakehouse as an Iceberg table could cost you... $10/hr? Bufstream does it. It's literally too good to be true. I spent 20 hours researching them. Here's their story (2 minute read) 🧵

Confluent runs tens of thousands of Kafka clusters across 93 regions in 3 clouds - AWS, GCP, and Azure. How do they manage all of that? Here are 13 innovative modifications they did to run Kafka smoothly at that scale. 👇

Nobody does data infrastructure like Uber does. Here are their numbers 👇

a new edition of 2 minute streaming is waiting patiently in your inbox probably the simplest and fastest explanation of AWS networking costs out there on the internet (plus a gift announced at the end) 🎁 ✅ blog.2minutestreaming.com

how do you read this? you pay $0.02 when you cross AZ. do you pay an extra $0.02 if it's going through a public IPv4? or does it imply you pay $0 when going cross-AZ though private ip? ipv6: do you pay an extra charge when going cross-VPC? or is it $0 cross-AZ same-VPC? 🤷‍♂️🤷‍♂️

Genius is making complex ideas simple, not making simple ideas complex. Over 70% of Fortune 500 companies have used Apache Kafka. At its core, it’s just a distributed commit log. A log (a.k.a. {write-ahead, commit, transaction} log) is a simple but efficient data structure:

The largest performance improvements often come the easiest. Here’s an Apache Kafka config tweak to increase your performance by 50% 🔥 (a 1-minute 🧵)

man I can't get it to post here without errors, so you'll have to go on the bird app if you want to see it. But I broke down the WarpStream story in tweet format: x.com/BdKozlovski/... I also explained very explicitly what my position is regarding the situation. So before you judge - 👀

The spiciest post I’ve published yet is out now. 🌶️ "The Brutal Truth About Kafka Cost Calculators" I also announced a new product that I’ve been heads down coding for the last two months. Interested? 👀 bigdata.2minutestreaming.com/p/the-bruta...

I'm exposing something about the Kafka industry tomorrow. I usually don't hype these up but tomorrow's newsletter edition will be the most impactful one I've released yet. It will be spicy. It will change how you think. You don't wanna miss it. 🌶️

Every data engineer is talking about S3’s new features from this reInvent. But can you remember all the others? Here is a small cheatsheet with AWS S3’s top 12 features to help you keep up👇

Looking at the API, it wouldn't be that hard to extend Kafka's Tiered Storage plugin to write to S3 Tables in an Iceberg format directly. The only question would be - where do you get the topic schema from? Which makes me question... why doesn't Kafka have first-class schema support?

More thoughts about the S3 Iceberg release in a new long-form piece, including: • whether we can call Iceberg the winner of the table format wars • how I calculate it 37% more expensive than regular S3 • where the lock-in may be • + lots of references! (link in reply)

Yesterday, AWS shook the data lake world by releasing two new S3 features that will forever cement its place there. 👑 Every data engineer must become familiar with them. A short thread on these game-changers 🧵 (2 minute read)

What people think Kafka compaction is: • unique keys ❌ What it actually is: • a retention strategy denoting how you preserve existing data 💡 A compacted topic is one where you have a more granular retention strategy. You're basically saying "retain at least one record per key".

Got a juicy Kafka letter ready to go out tomorrow Here's a leak: