Today, in the Data Streaming Journey, I'm sharing my experience of building data streaming products with the RAD stack: Rust, Arrow, DataFusion. www.streamingdata.tech/p/streaming-...

Really, Grammarly?

Introducing Iron Vector: native, columnar, vectorized, high-performance accelerator for Apache Flink SQL and Table API built on top of Rust, Arrow and DataFusion. Reduce your Flink compute cost by up to 2x or handle 2x more data with the same infrastructure.

Introducing Iron Vector: native, columnar, vectorized, high-performance accelerator for Apache Flink SQL and Table API built on top of Rust, Arrow and DataFusion. Reduce your Flink compute cost by up to 2x or handle 2x more data with the same infrastructure.

The benefit of working two jobs: when you're tired of one, you can switch to another.

This is too good not to share: www.scattered-thoughts.net/writing/sql-...

I was compelled to make this

arrow-avro: I'm an experimental library! me: ok, not a big deal me, three days later: oh, I see 😬

The ability to keep up with DataFusion upgrades is a skill. The project truly leverages "breaking changes are fine in a major version upgrade" rule. Just look at this: datafusion.apache.org/library-user... 😅

TIL DataDog has a somewhat usable FREE plan. Yes, free. DataDog. Free. I know. The metric retention is only one hour, but it's enough to do some loadtesting.

In the past few weeks, I fixed memory leaks, JVM GC thrashing, a deadlock and all sorts of concurrency issues 😅. However, the Iron Vector POC is now complete 😎. Running benchmarks now and starting on the V1 roadmap.

I'm glad I see almost no mentions of "Reverse ETL" anymore. Cause it's all just ETL. Or, to be precise, data integration.

A short post about the recent Aiven announcement regarding adding Iceberg support for Kafka www.streamingdata.tech/p/no-more-ex...

“$110M a year on AWS? I could do it for half!” Bro, your last infra project was making a cron job email you weather updates. Sit down. www.duckbillgroup.com/blog/figmas-...

Over the past few months, I've been building a new Irontools extension called Iron Vector. It's a native, columnar, vectorized runtime for Apache Flink SQL and Table API. It's built on top of Rust, Arrow and DataFusion.

Still, probably the most entertaining conference talk I watched. Worth the time www.youtube.com/watch?v=RZ4S...

As a person who's building a plugin system based on dynamic loading, Rust and FFI, this feels very refreshing www.redpanda.com/blog/multi-l.... Just use gRPC over a Unix socket! I'd love to see a performance comparison though, going over gRPC must slower, even with batching.

🤣😅😭

I've just caught someone posting AI-summarized versions of my streaming data newsletter (www.streamingdata.tech) on LinkedIn. Without any attribution, of course.

I’ve been saying this for a while: data streaming needs to be polyglot. I’m addressing this with Iron Functions: a WebAssembly-based runtime that enables writing #ApacheFlink transformations in several languages like TypeScript, Rust, and Go.

What's your getpocket.com export stats?

Last night 🤘🥁

New post! I wrote about malaise in the streaming space. I'm not sure what's going on, but I'm thinking part of it has to do with Kafka.

Current London 2025: Kafka, Flink, Iceberg, AI... and some drama. www.streamingdata.tech/p/current-lo...

At #WebSummit Vancouver. It’s nice to travel to a conference in 20 minutes with no flights involved 😀

Opening Bluesky this morning...

Just saw police deployed a spike strip to stop a car in London. Wild.

Arrived at #Current25

I understand the narrative around losing to someone using AI, but that's not the game I want to play. I'm working to be me; there is no competition.

Wrote up some notes about eng levels and AI skamille.medium.com/things-i-cur...

TIL you can stream ClickHouse data in Arrow format natively: clickhouse.com/docs/interfa...

How did we allow Jackson to become the default Java JSON library? 🫠

Our industry is constantly evolving and so are projects like #Debezium. Competition is good an all that but if you position your new shiny solution against existing tech, please be honest, authentic and truthful... real world example below👇

I launched a new company! 🛠️ Irontools is a suite of Apache Flink extensions that makes your Flink pipelines more efficient and unlocks new use cases. For example, it's now possible to create UDFs in TypeScript! Check the announcement post here: irontools.dev/blog/announc...

Setting up cross-platform build for a Rust project the first time 😅

Newsletter: I am sick and god damn tired of everybody pretending that generative AI is the next big thing. The media is complicit in accepting fantastical nonsense - both in the numbers put out by OpenAI and the silly jobs created by Anthropic - and it has to stop. www.wheresyoured.at/reality-check/

Movie you’ve watched more than six times using gifs. (“Hard mode” no Star Wars, Star Trek, or LOTR)

Incorporated a company today 😎

Just published: "Modern Data Streaming Challenges: Part 2". Thinking about developer experience. www.streamingdata.tech/p/streaming-...

If you are an aspiring founder out there, struggling to find the perfect name for your company, consider that having a name as stupid as "Supabase" didn't keep a good product from achieving a $2B valuation.

I've realized why I like prog metal: it's perfect for Pomodoro-style focus intervals! 😄

Chiming in with my analysis of Timely and Differential, as well as internal consistency concept: www.streamingdata.tech/p/attempting...

Flashback to 2020 when Gartner predicted tools like Bubble, Retool and other visual-like app programming solutions would be how most software development would be done by today Not how it turned out Hard to predict the future and experts often just guess! www.alphasoftware.com/blog/why-low...

Last night

Just published: "Modern Data Streaming Challenges: Part 1" www.streamingdata.tech/p/streaming-...

It's happening again! 🥁 Vancouver, April 1st. Great music: we're playing 80s and 90s rock hits: Radiohead, RHCP, Depeche Mode, etc. vanrockshows.com/roxy

Really curious what's behind this 🤔

HYTRADBOI ("Have you tried rubbing a database on it?", @hytradboi.bsky.social) is happening again! It's a really cool, forward-looking online event on everything databases. Highly recommend to check it out and attend. Happening on Feb 28, 9:00-15:00 PDT. www.hytradboi.com/2025

Just published: Exploring Apache DataFusion as a Foundation for Streaming Framework www.streamingdata.tech/p/exploring-...