CloudFlare's Distributed R2 SQL engine's is a pretty good exemplar of how to build a serverless database to process petabytes in seconds using Apache DataFusion and Apache Parquet blog.cloudflare.com/r2-sql-deep-...

I cannot say enough about DataFusion...in order to build an engine that considers spatial types at every level we needed to customize types, functions, optimizer rules, joins, Parquet pruning, and more. DataFusion not only made this possible but documented even the most obscure bits. So cool!

So Cool -- jcsherin added full text indexes into Parquet files using the techniques from our blog github.com/jcsherin/dat...

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen" Built in Rust with Apache DataFusion sedona.apache.org/latest/blog/...

We just published an easier to find list of all PMC and committers of Apache DataFusion , and it is quite a cool list of people and affiliations if I do say so myself 🤗 datafusion.apache.org/contributor-...

And we are also adding Geometry to the Rust parquet implementation . Huge thanks to @kylebarron.dev github.com/apache/arrow...

It was a great time on Monday at the @apachedatafusion.bsky.social meetup in NYC. We heard about distributed query plans, filter pushdown, geospatial support, and VegaFusion. More deets here github.com/apache/dataf...

One unfortunate aspect of new AI tools is they make it easier to generate large amounts of plausible looking code, which puts an even bigger burden on the reviewers, and reviewers are already the resource open source projects have the least of.

Dynamic Filters for TopK and Join queries landing in DataFusion 50.0.0: datafusion.apache.org/blog/2025/09...

What is LiquidCache in these slides: what-is-liquid-cache.xiangpeng.systems BTW @xiangpeng.systems is looking for some early adopters who want to be on the bleeding edge. Hit me up if interested

Recording of "Introduction to Variant in @ApacheParquet ": www.youtube.com/watch?v=nlOJ... Here are the slides: docs.google.com/presentation...

I've been helping our analytics team integrate our DataFusion-based query engine for Postgres into EDB Postgres Distributed and finally here's an end-to-end demo. You get HA Postgres plus seamless replication and DataFusion-based queries. This query turned out 6x faster than PG.

We want `brew install tpchgen-cli` to work, but that requires the project to be "popular" enough according to homebrew (30 forks, 30 watchers and 75 stars) We just need 6 more forks and 24 watchers. Can you help us out? github.com/clflushopt/t... Deets: github.com/clflushopt/t...

Discover why #RustLang is the right choice for building #LowLatency systems: not just for code’s performance, but also for productivity & developer joy. 🎧 Listen to the #InfoQ #podcast with Andrew Lamb for more insights: bit.ly/47kgmwU #AI #ProgrammingLanguages #Performance #Concurrency #DevEx

Thanks to @clflushopt.bsky.social, make massive TPCH datasets with tpchgen-cli 2.0: SF1000 (1TB raw, 220GB in @ApacheParquet ) in less than 10 mins (6m45s) on aging laptop Try it now: pip install tpchgen-cli tpchgen-cli --scale-factor 1000 --parts 100 --format=parquet github.com/clflushopt/t...

It is a common misconception that Parquet requires (slow) reparsing metadata and is limited to built in indexing structures. Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Parquet with @apachedatafusion.bsky.social datafusion.apache.org/blog/2025/08...

We are doing another DataFusion meetup in Boston Wednesday Nov 12, 2025: lu.ma/w9pw5rce

"EDB claimed the new engine, which pushes queries to open source @apachedatafusion.bsky.social , returned queries 30x faster than standard Postgres while tiering offloads cold transactional data to storage is up 18x more cost-efficient." www.theregister.com/2025/06/20/e...

@apachedatafusion.bsky.social 48.0.0 release. Spark Compatible functions, ORDER BY ALL, FFI for aggregates and window functions: datafusion.apache.org/blog/2025/07...

It is a common misconception that Apche Parquet files are restricted to basic statistics. Footer metadata and offset-based addressing permit user-defined index structures today. @apachedatafusion.bsky.social blog from Qi Zhi, Jigao Luo and myself explains how datafusion.apache.org/blog/2025/07...

Belated DataFusion 47.0.0 release blog: datafusion.apache.org/blog/2025/07...

Apahce Iceberg compaction in Rust via github.com/nimtable/ice.... (based on @apachedatafusion.bsky.social ). Thanks to Rising Wave CEO YingjunWu for the shout out

Apache Arrow Summit 25 is happening! Join us in person on October 2nd, in Paris (hosted by @pydataparis.bsky.social ). The Call For Proposal is open, submit your talks before July 26th: sessionize.com/arrow-summit...

Sweet VLDB paper from TUM (Mateusz Gienieczko / github.com/v0ldek) proposing extending Apache Parquet using user defined encodings (via WASM). Favorite image shows the ease of integrating into @apachedatafusion.bsky.social gienieczko.com/anyblox-paper

I am speaking at the #ApacheIceberg NYC Meetup on July 10th about Variant in #ApacheParquet which enable more efficient of processing semi structured data such as that found in JSON. lu.ma/95a5qys1

Quite a list of contributors already to the Rust Apache Parquet implementation of Variant (support for semi structured data). I was making some slides to explain what Variant is and made up a list I wanted to share. The feature will be amazing github.com/apache/arrow...

I publicly apologize for snapping today with @andypavlo.bsky.social @pateljm.bsky.social. "You need to have a push based scheduler to do ..." TUM / DuckDB(CWI) created group-think in Databases where push schedulers are required, ClickHouse, Spark, DataFusion, etc not withstanding 🤦

New blog post about cooperative scheduling using tokio and Rust async, and how cancellation works in @apachedatafusion.bsky.social datafusion.apache.org/blog/2025/06...

Two new (reposted) blogs about Optimizing SQL and DataFrames: datafusion.apache.org/blog/2025/06... datafusion.apache.org/blog/2025/06...

This is so cool -- an example of embedding a special index (a DistinctValues index no less) inside a Paruqet file: github.com/apache/dataf... (coming in DataFusion 49.0.0) @apachedatafusion.bsky.social

Here is another product built on @apachedatafusion.bsky.social : blog.cloudflare.com/logexplorer-...

Intro to @apachedatafusion.bsky.social : Technology, Community and Not Quite Enough Time: www.youtube.com/watch?v=3per...

Hot off the presses: I recorded my talk from this week about Accelerating Query Performance of Apache Parquet using Specialized Indexes: youtu.be/74YsJT1-Rdk

Reminder: San Francisco @ApacheDataFusio meetup tomorrow: lu.ma/uuxd443e

Generating TPCH SF 10 (almost SF-100) in Apache Parquet format on 4GB Raspberry pi whateverforever.computer/blog/tpchgen... Also found it could fry an 🍳

We are organizing another @apachedatafusion.bsky.social meetup in San Francisco that coincides with Data and AI. Come hear about Sail, Embucket and Apache Parquet filtering. Signup: lu.ma/uuxd443e

ClickBench keeps me convinced that Parquet can be quite fast. There is only a 2.3x performance difference vs @duckdb 's own format and unoptimized parquet: tinyurl.com/5aexvsfw I am surprised that the (closed source) Umbra only reports 3.3x faster than DuckDB on parquet

New DataFusion is looking pretty sweet performance wise. Thanks to github.com/MrPowers for running the numbers More details github.com/apache/dataf...

DataFusion -> Open Telemetry integration, courtesy of Geoffrey Claude at @datadoghq.com github.com/datafusion-c...

InfluxData Staff Engineer @andrewlamb1111.bsky.social breaks down how @apachedatafusion.bsky.social lets you build database prototypes fast and innovate faster. Get a behind-the-scenes look at modern database stacks. 🎧 bit.ly/4jypZLB #InfluxDB

@apachedatafusion.bsky.social meetup June 9, in San Francisco. f you are in town for Data and AI summit perhaps you can join us lu.ma/uuxd443e

Huge thanks to @krisajenkins.bsky.social for the wonderful discussion about @apachedatafusion.bsky.social on Developer Voices: pod.link/developer-vo.... Kris's smooth voice and wide ranging knowledge made it most pleasurable.

@apachedatafusion.bsky.social is basically a dream: can a self-organizing group of database enthusiasts build an open contribution engine that rivals the best financed alternatives. We are not done yet but the signs are promising.

@apachedatafusion.bsky.social supports a fully extensible range of user defined window functions as well as scalar, aggregate and table functions. Aditya Singh Rathore's new blog has more: datafusion.apache.org/blog/2025/04...

MIT following Harvard's lead here

Variant is coming soon to @ApacheParquet in Rust . Huge thanks to @db.cs.cmu.edu for getting the process started in @ApacheArrow with a great draft PR to kick off Variant support: github.com/apache/arrow... 🙏🙏🙏 Thank you

Worlds Fastest TPCH Data Generator, courtesy of @ApacheDataFusio 's community. Scale Factor 100 in under 2 minutes on Macbook air. Open Source, no dependency Rust. Inspiration hat tip to @db.cs.cmu.edu and @wslim.bsky.social datafusion.apache.org/blog/2025/04... www.youtube.com/watch?v=UYIC...

Example of the kind of low level performance obsession ApacheDataFusion is attracting: github.com/apache/dataf.... Huge thanks to acking-you and everyone else who participated

We're pleased to announce that Apache DataFusion in Python 46.0.0 is released! Since the last announcement post we've had a lot of great features and new contributors. Please check out the blog post with details. datafusion.apache.org/blog/2025/03... #DataFusion #Python #DataFrame #PyData #Apache

DataFusion Python 46 release: datafusion.apache.org/blog/2025/03...