I wrote a bit about the "demand transform" for Datalog (and other similar languages): github.com/frankmcsherr... The idea of the transform is that rather than eagerly produce all the data you might ever need, you can start from the seed crystals of concrete values and data that might yield output.

If anyone ever told you Datalog is elegant, they probably haven't used datatoad yet. The evenness condition is an interesting example of relation-rather-than-function, though!

I went through and validated the output counts (numbers of facts) for most of the datatoad outputs. Some were hard to validate because datatoad sneaked in on my laptop but the other systems paged too hard, but everything seems correct. github.com/frankmcsherr...

I put up a new post about work that I am perhaps irrationally excited about: the combination of worst-case optimal joins with relational programming. github.com/frankmcsherr... The high-order bits are that relational programming is very neat, and it fits great with modern (WCO) relational joins.

I wrote a post evaluating datatoad using the framework from the recent FlowLog paper: github.com/frankmcsherr.... It turns out it does well in some cases, worse in others, and has already improved by having other folks shine a light on its limitations by choosing problems and datasets I ignored!

I wrote about datatoad as a "worst-case optimal Datalog", which .. perhaps doesn't typecheck. A previous result on streaming worst-case optimal joins seems to connect, and swapping "streaming" for "iterative" gives something that may be syntactically correct catnip. github.com/frankmcsherr...

Getting crisper; now within ~1s (14.8s v 13.7s) of my hand-optimized plan that uses only binary joins, but without any plan hints. Erm, mostly. Hoping to bypass it, as a win for WCO joins. :D

I wrote a little bit about worst-case optimal joins in datatoad, support for which has just landed although there is a bit of tidying still to do before they are as crisp as the non-wco joins. github.com/frankmcsherr...

A short note on how one can perform joins of sums by accumulating at a point halfway through the join. Neither the input nor the output of the join, but smaller intermediate terms that show up when you peer closer at what a join does internally. github.com/frankmcsherr...

I wrote a bit about datatoad going "full columnar". All operations are now columnar; no rows are ever formed; everything is column-at-a-time. github.com/frankmcsherr... A bunch of interesting (to me) algorithms, and also some performance regressions, but then clawing back. I learned things!

Looking forward to this!

New from Materialize: Cloud M.1 Clusters Run 3x larger workloads with the same low latency and predictable performance—thanks to intelligent data spilling and expanded capacity. Learn more: bit.ly/3L12oH2

Datatoad check-in: this time including some recent progress on columnar joins (good news: faster). Though, it also tries to roll up a bit of the sprawl of content I've scribbled, which increasingly feels like it needs some more careful curation to be helpful. github.com/frankmcsherr...

Good news on the Datalog front: v1 of "columnar joins" seem to work, and resulted in a 20% improvement (from 9.5s to 7.5s, for the joins of a reference workload). Still more gains from tightening it up, and potentially from columnar sorting, but I'll take a swing at writing things up tomorrow!

What a difference an allocator makes! This is the same Rust program first using the system allocator, and then using mimalloc. About 100MB of working set in both cases, just .. apparently it pilots the system allocator to some horrible behavior. Obviously going to start using mimalloc from now on.

Welcome Frank McSherry @frankmcsherry.bsky.social to Sync Conf 2025. Pioneer of sync technology, inventor of Differential Dataflow, and founder of @materialize.com, Frank will trace the evolution of sync and stream processing.

Highlighting some of my team's recent work: We've changed Materialize to use swap instead of memory-mapped files, with nice performance and efficiency improvements.

We’ve released a major improvement to our memory spilling infrastructure: Materialize now uses swap to scale SQL workloads beyond RAM. ✅ Faster hydration ✅ Efficient memory utilization ✅ Bigger workloads supported Full post from antiguru.bsky.social → bit.ly/46EF2iJ

Very excited to bring some column-orientation to timely and differential. At least, removing baked in row-orientation in timely, and actual column-orientation in differential, with a bunch of cool learnings from the datatoad work. I hope. We'll see. :D

We just released Timely Dataflow 0.24! Here are some exciting changes from @frankmcsherry.bsky.social and myself. The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below. github.com/TimelyDatafl...

I have a trip coming up, and I'm hoping to find some content to read about the implementations of (ideally interpreted) array languages. I'm on an interpreter kick, and armed with a bunch of column-oriented libraries. Any tips, drop a reply!

I wrote a bit about datatoad's columnar logic for relational operators. At least, for union, intersection, antijoins, and semijoins. It turns out the joins are all easy; it's projection that is hard, of all things. Go figure. github.com/frankmcsherr...

Notion's new offline support is based on our rich text CRDT research x.com/ivanhzhao/st...

If you are in SF in November, I'll be speaking at syncconf.dev (@syncconf.bsky.social)! It's an excellent confluence of all things up-to-data. Architectures like MZ at the backend, connected via sync engines, and front ends that don't waste anyone's time waiting on database queries.

In Datalog news: I had given up on getting (compiled) datafrog numbers for the "alias analysis" problem, because it is tedious to write. But thanks to an anonymous benefactor, it was coded up and we can now make a comparison between compiled datafrog and interpreted datatoad, on the same problem!

I wrote about the projects done at Materialize’s recent hackathon. Many very cool projects, and also one that I worked on; take a read! materialize.com/blog/spring_...

A neat new Materialize post from our QA department on speeding up CI. materialize.com/blog/speedin...

Datalog weekend: we graduate to queries with cyclic rules, non-binary relations, and generally more interesting behavior. In particular, we're going to compare ourselves against interpreted Soufflé; a standard reference point! How does interpreted datatoad compare? github.com/frankmcsherr...

We have a new @materialize.com post up, this time about pushing selection predicates into our persistence layer. Materialize hybridizes batch and streaming computation (it does both, regularly), and draws on the best optimizations of each (in this case, CDWs). materialize.com/blog/how-fil...

Nice that the Bluesky firehose is now becoming a live dataset on which to demo streaming databases

We have a weekend Datalog update: datatoad now comes with tries! I took the long route to get here, and have a bit of other work in the pipeline as part of all this. But I'm happy to report that memory went down by another 2x (expected), and runtime went down by ~1.6x. github.com/frankmcsherr...

We have a new blog post up at @materialize.com about analyzing the Bluesky firehose (Jetstream, really) through Materialize. You can grab a copy of the community edition of MZ and follow along, or invent your own ways of looking at the data, live! materialize.com/blog/analyzi...

I had a great time at Kris's workshop in May! Lots of inspiring talks and discussions. He has posted a mega-video of all the recorded talks, I highly recommend checking some of them out if you're into Datalog, logic programming, or incrementalization. There is even some e-graph stuff in there!

Folks! The workshop at Minnowbrook that prompted the datatoad work has put up the videos from the talk. NINE HOURS! Of logic programming content! I liked the talks; they are chaptered; you can see @mwillsey.com talk about e-graphs; why haven't you clicked yet??? www.youtube.com/watch?v=3ec9...

Word is getting around.

I refreshed a blog post draft on streaming the Bluesky firehose through @materialize.com. Some experience tidied up the examples, made things a bit more efficient, and told a different story (now with less cloture). More in the near future, as we put a front end on it! github.com/frankmcsherr...

Today's Datalog update: I attempt to close the gap between interpreted (datatoad) and compiled (datafrog) Datalog. I do not succeed! But we do get a ~3x performance bump for our reference workload, with line of sight on further improvements (and another 3x gap to close). github.com/frankmcsherr...

A new post by @nate-stewart.bsky.social explains how an always-up-to-date "digital twin" of your part of the real world is increasingly important as the folks who want to interact with you and your stuff expand beyond just humans. materialize.com/blog/agents-...

Up next in Datalog: streamlining our representation of facts and terms. We've already taken 50GB of state down to 5GB with program optimizations; let's take it down another 10x with more thoughtful data structures. github.com/frankmcsherr...

Ongoing Datalog work; this week: implementing the execution of e-graph optimized plans. I also added "teaser" content at the bottom, associated with each of the "todo" sections around spilling to disk, scaling out, etc. Let me know if any sound especially interesting! github.com/frankmcsherr...

Next in the Datalog series: writing an optimizer. I used e-graphs to get a feel for how they work, and .. debugging e-graph rules is a lot different from debugging Rust. I really like the idea of forcing myself to use it, and teaching it what rules I think I know. github.com/frankmcsherr...

Materialize 25.2 is here! New features include live freshness reports for all your views, 2.5x faster data product deployment times, and native SQL Server support. See how these updates can help streamline your operations: bit.ly/44i2hg2

The v25.2 release of @materialize.com Self-Managed just went out. Performance, productivity, observability, integrations ... lots of good new stuff in there to try out if you've been thinking about taking Materialize out for a self-managed spin. materialize.com/blog/materia...

An update to the interactive Datalog post: some experimentation! Two program analysis tasks, pretty friendly ones, that ended up telling imo a pretty nice story about Datalog optimization. I'm looking for more examples (with data) to try out and break things; lmk! github.com/frankmcsherr...

Mysteriously, the blog post ended up being longer (in lines) than the code. Just over 1k for the post, and just under for the code. Maybe just because a lot of the code made it into the post, but also .. less code and more explanation ftw, imo.

A new blog post on an interactive Datalog in Rust! Early days, but 1. nice and interactive, and 2. not less performant than my prior attempt (datafrog) when you use strings as the literal type (vs u32). More to come, as it's partially meant as a playground for me. github.com/frankmcsherr...

Current side project: an interactive Datalog in Rust. Should be a few blog posts about it coming, once I make sure it isn't utterly terrible. It may be a while though, with that constraint.

A new blog post! You can implement worst-case optimal joins in databases that only support trees of binary joins. In fact, we did it ten years ago and didn't notice. Oops. Binary joins are not the problem; it's query plans that use *only* binary joins (and not counts). github.com/frankmcsherr...

Very neat post up from @materialize.com engineers on finding and fixing unsoundness in Rust's mpsc channels. We had quite a few grim stack traces implicating timely, but that turned out (phew) to be Rust memory corruption rather than timely glitches. materialize.com/blog/rust-co...

Another blog post up, this time on the coordination primitive needed by timely's progress tracking protocol. We implement this with point-to-point channels, but it could be substantially more efficient. The full design space isn't entirely clear to me, though. github.com/frankmcsherr...