Another brand new new feature is the R2 data catalog: blog.cloudflare.com/cloudflare-d... Build something with Pipelines and R2 SQL. I suggest receiving OpenTelemetry data and then surfacing that in a web app (logs should be fairly straightforward), but there are tons of uses for this.

It's early, but I'm excited about direction that the Cloudflare Data Platform is taking. Trying to set up similar pipelines on other clouds would typically be $$$ and take tons of expertise. Managing kafka and multiple services for ingestion, compaction, etc blog.cloudflare.com/cloudflare-d...

The news is finally out! Cloudflare has a Data Platform! We're starting with serverless streaming pipelines (powered by arroyo), a managed Iceberg Catalog, and a new distributed SQL engine built on top of DataFusion

Reminder: San Francisco @ApacheDataFusio meetup tomorrow: lu.ma/uuxd443e

Cloudflare is at Snowflake Summit in San Francisco this week! Swing by our booth 2605 to chat about the new Cloudflare R2 Data Catalog and how it can make your data management and analytics easier!

Next Monday after the Snowflake Summit keynote! Hang out on our beautiful roof with other cool data folks, and hear some great speakers from LanceDB, @mooncakelabs.bsky.social, Eventual, Marimo, Bobsled, and @cloudflare-dev.bsky.social! lu.ma/dbq1hfij

Ok, y'all. This took me several weeks and a ton of help from @frankmcsherry.bsky.social and @lalithsuresh.bsky.social. I dug into timely dataflow, differential dataflow, and DBSP to get you up to speed on IVM engines and materialized views. Enjoy!

I’m only a week into life at @cloudflare-dev.bsky.social but already amazed by how much of Cloudflare is built _on_ Cloudflare. I’d never have guessed you could get so far with just workers + durable objects!

Arroyo is joining @cloudflare.social! We're bringing Arroyo to the Developer Platform as a serverless stream processing system, and will also remain open-source and self-hostable. www.arroyo.dev/blog/arroyo-...

Couple of big announcements from @cloudflare.social today for folk in #dataBS: * Acquisition of Arroyo, launch of Pipelines for streaming ingestion: blog.cloudflare.com/cloudflare-a... * Launch of R2 Data Catalog—a managed Apache Iceberg catalog for R2 blog.cloudflare.com/r2-data-cata...

Arroyo 0.14.0 is now available, including new lookup joins, support for nested updating aggregates, struct types, new syntax, and a bunch of improvements and fixes: www.arroyo.dev/blog/arroyo-...

I know by month 2 we're all inured to this stuff, but this is a beyond crazy mix of incompetence and illegality www.theatlantic.com/politics/arc...

Arroyo is sitting at 3,999 stars... who's going to put us over the top github.com/ArroyoSystem...

You'd think that the key to being a fast streaming engine is like clever join algorithms, but it's mostly just being really good at JSON. Arroyo uses Arrow and the arrow-rs JSON decoder along with some streaming extensions. I think it's pretty cool, so I wrote up a long explanation of how it works

Our team at Arroyo recently needed to rebuild our (very ad-hoc) analytics infra to account for our growth. We spent some time working out the best way to set up a near-real-time data lake today, and ended up with a pretty sweet approach we're calling the LOAD stack: www.arroyo.dev/blog/buildin...

Arroyo 0.13.0 is now available! This one includes some big improvements to the core engine a (including the operator chaining work I wrote about previously: bsky.app/profile/mica...) and a bunch of other features. All the details on our blog: www.arroyo.dev/blog/arroyo-...

Is #DataBS interested in the internals of streaming engines? The next release of arroyo.dev (0.13) has a new feature in the core dataflow called operator chaining which gets at some of the interesting details of how these systems work. So let’s dive in to streaming dataflow 🧵

nothing like a potential natural disaster to bring us all together here 🤗

Sad to see Redis Labs burning whatever shreds of credibility they still had with the open source community. Making money as an open source co is hard but there has to be better ways than this github.com/redis-rs/red...

Happy new Jepsen Report Day to all who celebrate! jepsen.io/analyses/buf... Confirms my priors that almost no one should be directly calling the Kafka client today—use a stream processing engine for data use cases or a durable execution engine for applications.

If you missed #p99conf last week, talks are now available to stream on YouTube. I spoke about the design decisions that went into Arroyo's incredible performance: youtube.com/watch?v=7H4C... Come for the Rust hot takes, stay for my terrible hand-drawn architecture diagrams 😅

Guess we're all here now 👋