What a difference an allocator makes! This is the same Rust program first using the system allocator, and then using mimalloc. About 100MB of working set in both cases, just .. apparently it pilots the system allocator to some horrible behavior. Obviously going to start using mimalloc from now on.

Welcome Frank McSherry @frankmcsherry.bsky.social to Sync Conf 2025. Pioneer of sync technology, inventor of Differential Dataflow, and founder of @materialize.com, Frank will trace the evolution of sync and stream processing.

Highlighting some of my team's recent work: We've changed Materialize to use swap instead of memory-mapped files, with nice performance and efficiency improvements.

We’ve released a major improvement to our memory spilling infrastructure: Materialize now uses swap to scale SQL workloads beyond RAM. ✅ Faster hydration ✅ Efficient memory utilization ✅ Bigger workloads supported Full post from antiguru.bsky.social → bit.ly/46EF2iJ

Very excited to bring some column-orientation to timely and differential. At least, removing baked in row-orientation in timely, and actual column-orientation in differential, with a bunch of cool learnings from the datatoad work. I hope. We'll see. :D

We just released Timely Dataflow 0.24! Here are some exciting changes from @frankmcsherry.bsky.social and myself. The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below. github.com/TimelyDatafl...

I have a trip coming up, and I'm hoping to find some content to read about the implementations of (ideally interpreted) array languages. I'm on an interpreter kick, and armed with a bunch of column-oriented libraries. Any tips, drop a reply!

I wrote a bit about datatoad's columnar logic for relational operators. At least, for union, intersection, antijoins, and semijoins. It turns out the joins are all easy; it's projection that is hard, of all things. Go figure. github.com/frankmcsherr...

Notion's new offline support is based on our rich text CRDT research x.com/ivanhzhao/st...

If you are in SF in November, I'll be speaking at syncconf.dev (@syncconf.bsky.social)! It's an excellent confluence of all things up-to-data. Architectures like MZ at the backend, connected via sync engines, and front ends that don't waste anyone's time waiting on database queries.

In Datalog news: I had given up on getting (compiled) datafrog numbers for the "alias analysis" problem, because it is tedious to write. But thanks to an anonymous benefactor, it was coded up and we can now make a comparison between compiled datafrog and interpreted datatoad, on the same problem!

I wrote about the projects done at Materialize’s recent hackathon. Many very cool projects, and also one that I worked on; take a read! materialize.com/blog/spring_...

A neat new Materialize post from our QA department on speeding up CI. materialize.com/blog/speedin...

Datalog weekend: we graduate to queries with cyclic rules, non-binary relations, and generally more interesting behavior. In particular, we're going to compare ourselves against interpreted Soufflé; a standard reference point! How does interpreted datatoad compare? github.com/frankmcsherr...

We have a new @materialize.com post up, this time about pushing selection predicates into our persistence layer. Materialize hybridizes batch and streaming computation (it does both, regularly), and draws on the best optimizations of each (in this case, CDWs). materialize.com/blog/how-fil...

Nice that the Bluesky firehose is now becoming a live dataset on which to demo streaming databases

We have a weekend Datalog update: datatoad now comes with tries! I took the long route to get here, and have a bit of other work in the pipeline as part of all this. But I'm happy to report that memory went down by another 2x (expected), and runtime went down by ~1.6x. github.com/frankmcsherr...

We have a new blog post up at @materialize.com about analyzing the Bluesky firehose (Jetstream, really) through Materialize. You can grab a copy of the community edition of MZ and follow along, or invent your own ways of looking at the data, live! materialize.com/blog/analyzi...

I had a great time at Kris's workshop in May! Lots of inspiring talks and discussions. He has posted a mega-video of all the recorded talks, I highly recommend checking some of them out if you're into Datalog, logic programming, or incrementalization. There is even some e-graph stuff in there!

Folks! The workshop at Minnowbrook that prompted the datatoad work has put up the videos from the talk. NINE HOURS! Of logic programming content! I liked the talks; they are chaptered; you can see @mwillsey.com talk about e-graphs; why haven't you clicked yet??? www.youtube.com/watch?v=3ec9...

Word is getting around.

I refreshed a blog post draft on streaming the Bluesky firehose through @materialize.com. Some experience tidied up the examples, made things a bit more efficient, and told a different story (now with less cloture). More in the near future, as we put a front end on it! github.com/frankmcsherr...

Today's Datalog update: I attempt to close the gap between interpreted (datatoad) and compiled (datafrog) Datalog. I do not succeed! But we do get a ~3x performance bump for our reference workload, with line of sight on further improvements (and another 3x gap to close). github.com/frankmcsherr...

A new post by @nate-stewart.bsky.social explains how an always-up-to-date "digital twin" of your part of the real world is increasingly important as the folks who want to interact with you and your stuff expand beyond just humans. materialize.com/blog/agents-...

Up next in Datalog: streamlining our representation of facts and terms. We've already taken 50GB of state down to 5GB with program optimizations; let's take it down another 10x with more thoughtful data structures. github.com/frankmcsherr...

Ongoing Datalog work; this week: implementing the execution of e-graph optimized plans. I also added "teaser" content at the bottom, associated with each of the "todo" sections around spilling to disk, scaling out, etc. Let me know if any sound especially interesting! github.com/frankmcsherr...

Next in the Datalog series: writing an optimizer. I used e-graphs to get a feel for how they work, and .. debugging e-graph rules is a lot different from debugging Rust. I really like the idea of forcing myself to use it, and teaching it what rules I think I know. github.com/frankmcsherr...

Materialize 25.2 is here! New features include live freshness reports for all your views, 2.5x faster data product deployment times, and native SQL Server support. See how these updates can help streamline your operations: bit.ly/44i2hg2

The v25.2 release of @materialize.com Self-Managed just went out. Performance, productivity, observability, integrations ... lots of good new stuff in there to try out if you've been thinking about taking Materialize out for a self-managed spin. materialize.com/blog/materia...

An update to the interactive Datalog post: some experimentation! Two program analysis tasks, pretty friendly ones, that ended up telling imo a pretty nice story about Datalog optimization. I'm looking for more examples (with data) to try out and break things; lmk! github.com/frankmcsherr...

Mysteriously, the blog post ended up being longer (in lines) than the code. Just over 1k for the post, and just under for the code. Maybe just because a lot of the code made it into the post, but also .. less code and more explanation ftw, imo.

A new blog post on an interactive Datalog in Rust! Early days, but 1. nice and interactive, and 2. not less performant than my prior attempt (datafrog) when you use strings as the literal type (vs u32). More to come, as it's partially meant as a playground for me. github.com/frankmcsherr...

Current side project: an interactive Datalog in Rust. Should be a few blog posts about it coming, once I make sure it isn't utterly terrible. It may be a while though, with that constraint.

A new blog post! You can implement worst-case optimal joins in databases that only support trees of binary joins. In fact, we did it ten years ago and didn't notice. Oops. Binary joins are not the problem; it's query plans that use *only* binary joins (and not counts). github.com/frankmcsherr...

Very neat post up from @materialize.com engineers on finding and fixing unsoundness in Rust's mpsc channels. We had quite a few grim stack traces implicating timely, but that turned out (phew) to be Rust memory corruption rather than timely glitches. materialize.com/blog/rust-co...

Another blog post up, this time on the coordination primitive needed by timely's progress tracking protocol. We implement this with point-to-point channels, but it could be substantially more efficient. The full design space isn't entirely clear to me, though. github.com/frankmcsherr...

New blog post up detailing the steps to get a new timely/differential dataflow up and running in WASM in your browser. Pretty short read, and should be easy to follow along at home if you are keen. github.com/frankmcsherr... Of course, let me know if you have questions, or it doesn't work for you.

Fun dataflow news: prompted initially by @oliwessels.bsky.social, we've finally merged changes into dataflows timely and differential that allow programs that use them to be built into WASM, and run in the browser.

PhD Timeline xkcd.com/3081

Ok, y'all. This took me several weeks and a ton of help from @frankmcsherry.bsky.social and @lalithsuresh.bsky.social. I dug into timely dataflow, differential dataflow, and DBSP to get you up to speed on IVM engines and materialized views. Enjoy!

The administration is a perfect storm of dumb people who enjoy inflicting pain and are addicted to psychotic cruelty and quadrupling down. It ends in total ruin if you don't impeach him.

We've shipped a new version of timely dataflow (crates.io/crates/timely) which removes some potentially quadratic behavior. Scopes and operators would take resources proportional to #inputs x #outputs, even with sparse interior connectivity. This is now down to linear in inputs + input-output paths.

I just put up a video of me using @materialize.com to ingest and analyze live Bluesky data. Starts with a Javascript bridge to get the data in, and then builds up various live views using SQL. Caveat: it's a live demo with live data. Eek! www.youtube.com/watch?v=T6cY...

This guy got me through Sekiro (Guardian Ape in one; flex), but also turns out to be more stand-up than I would have imagined, more than most folks with a platform.

The first AT Protocol conference is happening March 22 & 23rd in Seattle! If you’re a developer interested in building on the protocol, hope to see you there.

Bluesky is the place for science.

Materialize for Everyone: Now Self-Managed & Free 💡 Self-Managed – Deploy in your own cloud, full control. 💡 Community Edition – Production-grade Materialize, free forever. Real-time data is now open to all. Learn more: bit.ly/43Bg8iQ

To keep up with the demands of AI projects your teams need to transform, deliver, and act on fast changing data no matter where it resides. Today, we’re making that possible by moving Self-Managed @materialize.com out of beta and introducing Materialize Community Edition (CE).

I'm very excited to share that @materialize.com now has a Self-Managed option, as well as a Community Edition license that lets you play around with the exact same bits but for free. materialize.com/blog/materia...

All the videos are up at www.hytradboi.com/2025#program now.