One fun nugget from the Boston @apachedatafusion.bsky.social meetup on Wednesday: DataDog reports they run 68+million queries per hour with DataFusion

Here is a nice examination of the benefits of building new systems using the extensibility of @apachedatafusion.bsky.social vs other systems. www.bauplanlabs.com/post/duck-hu...

Excited to be one of the attendees and present our work on the DataFusion-powered SedonaDB alongside a great lineup of talks! If you're in the Boston area come and say hi!

"if you want to go fast, go alone; If you want to go far, go together" New Apache Parquet Community page is up: parquet.apache.org/community/

We are holding the next Apache DataFusion meetup next Wednesday Nov 12 in Boston. lu.ma/w9pw5rce

If anyone wants to know why Xiangpeng Hao is a great mentor, they can read this response: github.com/XiangpengHao...

New version of Rust Apache Arrow and Apache Parquet is out -- includes new new metadata parser, new avro reader, geometry and variant support 🤯 arrow.apache.org/blog/2025/10...

I have heard from 3 people/projects in the last three days they are considering forks of iceberg-rust. I filed a ticket to see if we can figure out how to consolidate efforts: github.com/apache/icebe...

Apache DataFusion's policy for AI assisted contribution: AI is great, but not AI dumps: maintainers could finish the task faster by using AI directly, and the submitters gain little knowledge when acting as a pass through AI proxy. datafusion.apache.org/contributor-...

Some Apache Parquet nerd humor for Friday afternoon lists.apache.org/thread/36rdg...

We made Apache Parquet metadata parsing 3x-9x faster in the latest release of the Rust implementation arrow.apache.org/blog/2025/10...

Today's Future Data Systems Seminar Speaker: Ian Cook (@ian.columnar.tech) will present @columnar.tech's work on Apache Arrow's database connectivity API (ADBC). ADBC is available in modern DBMSs. Zoom talk open to public at 4:30pm ET. YouTube video available after: db.cs.cmu.edu/events/futur...

More Products built with Apache DataFusion: Palantir Foundry's Pipeline Builder www.palantir.com/docs/foundry...

Prateek Gaur and co at Snowflake reproduced the (great) results for the ALP encoding algorithm from CWI / Azim Afroozeh / Peter Boncz ALP achieves ZSTD levels of compression and much faster decode. We are discussing adding it to @ApacheParquet: lists.apache.org/thread/tjtln...

The talk on Votex @db.cs.cmu.edu youtube.com/watch?v=zyn_... is a great one. I think it would also be interesting to hear a counterpoint about Apache Parquet that explains actual technical details of that format, the Cathedral vs Bizzaar management, options with Metadata, etc

Our new thrift parser in the Rust Apache Parquet implementation is a 🎁 that keeps on giving performance wise 🚀 github.com/apache/arrow... We are also working on a blog post that has a deeper explanation

Yesterday I learned about the SpatialBench from Sedona github.com/apache/sedon... Which they based on the tpchgen-rs project from @clflushopt.bsky.social github.com/clflushopt/t... (BTW I a still looking for some more github watchers on tpchgen-rs so I can get it on homebrew)

BTW if anyone wants a good intro to database storage / Log structured storage (aka LSM trees) @db.cs.cmu.edu lecture this fall is a good one: www.youtube.com/watch?v=2_sT...

It starts: github.com/clflushopt/t... @clflushopt.bsky.social is going to make the worlds fastest tpc-ds generator

Apache DataFusion 50 is released. Read all about it here: datafusion.apache.org/blog/2025/09...

CloudFlare's Distributed R2 SQL engine's is a pretty good exemplar of how to build a serverless database to process petabytes in seconds using Apache DataFusion and Apache Parquet blog.cloudflare.com/r2-sql-deep-...

I cannot say enough about DataFusion...in order to build an engine that considers spatial types at every level we needed to customize types, functions, optimizer rules, joins, Parquet pruning, and more. DataFusion not only made this possible but documented even the most obscure bits. So cool!

So Cool -- jcsherin added full text indexes into Parquet files using the techniques from our blog github.com/jcsherin/dat...

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen" Built in Rust with Apache DataFusion sedona.apache.org/latest/blog/...

We just published an easier to find list of all PMC and committers of Apache DataFusion , and it is quite a cool list of people and affiliations if I do say so myself 🤗 datafusion.apache.org/contributor-...

And we are also adding Geometry to the Rust parquet implementation . Huge thanks to @kylebarron.dev github.com/apache/arrow...

It was a great time on Monday at the @apachedatafusion.bsky.social meetup in NYC. We heard about distributed query plans, filter pushdown, geospatial support, and VegaFusion. More deets here github.com/apache/dataf...

One unfortunate aspect of new AI tools is they make it easier to generate large amounts of plausible looking code, which puts an even bigger burden on the reviewers, and reviewers are already the resource open source projects have the least of.

Dynamic Filters for TopK and Join queries landing in DataFusion 50.0.0: datafusion.apache.org/blog/2025/09...

What is LiquidCache in these slides: what-is-liquid-cache.xiangpeng.systems BTW @xiangpeng.systems is looking for some early adopters who want to be on the bleeding edge. Hit me up if interested

Recording of "Introduction to Variant in @ApacheParquet ": www.youtube.com/watch?v=nlOJ... Here are the slides: docs.google.com/presentation...

I've been helping our analytics team integrate our DataFusion-based query engine for Postgres into EDB Postgres Distributed and finally here's an end-to-end demo. You get HA Postgres plus seamless replication and DataFusion-based queries. This query turned out 6x faster than PG.

We want `brew install tpchgen-cli` to work, but that requires the project to be "popular" enough according to homebrew (30 forks, 30 watchers and 75 stars) We just need 6 more forks and 24 watchers. Can you help us out? github.com/clflushopt/t... Deets: github.com/clflushopt/t...

Discover why #RustLang is the right choice for building #LowLatency systems: not just for code’s performance, but also for productivity & developer joy. 🎧 Listen to the #InfoQ #podcast with Andrew Lamb for more insights: bit.ly/47kgmwU #AI #ProgrammingLanguages #Performance #Concurrency #DevEx

Thanks to @clflushopt.bsky.social, make massive TPCH datasets with tpchgen-cli 2.0: SF1000 (1TB raw, 220GB in @ApacheParquet ) in less than 10 mins (6m45s) on aging laptop Try it now: pip install tpchgen-cli tpchgen-cli --scale-factor 1000 --parts 100 --format=parquet github.com/clflushopt/t...

It is a common misconception that Parquet requires (slow) reparsing metadata and is limited to built in indexing structures. Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Parquet with @apachedatafusion.bsky.social datafusion.apache.org/blog/2025/08...

We are doing another DataFusion meetup in Boston Wednesday Nov 12, 2025: lu.ma/w9pw5rce

"EDB claimed the new engine, which pushes queries to open source @apachedatafusion.bsky.social , returned queries 30x faster than standard Postgres while tiering offloads cold transactional data to storage is up 18x more cost-efficient." www.theregister.com/2025/06/20/e...

@apachedatafusion.bsky.social 48.0.0 release. Spark Compatible functions, ORDER BY ALL, FFI for aggregates and window functions: datafusion.apache.org/blog/2025/07...

It is a common misconception that Apche Parquet files are restricted to basic statistics. Footer metadata and offset-based addressing permit user-defined index structures today. @apachedatafusion.bsky.social blog from Qi Zhi, Jigao Luo and myself explains how datafusion.apache.org/blog/2025/07...

Belated DataFusion 47.0.0 release blog: datafusion.apache.org/blog/2025/07...

Apahce Iceberg compaction in Rust via github.com/nimtable/ice.... (based on @apachedatafusion.bsky.social ). Thanks to Rising Wave CEO YingjunWu for the shout out

Apache Arrow Summit 25 is happening! Join us in person on October 2nd, in Paris (hosted by @pydataparis.bsky.social ). The Call For Proposal is open, submit your talks before July 26th: sessionize.com/arrow-summit...

Sweet VLDB paper from TUM (Mateusz Gienieczko / github.com/v0ldek) proposing extending Apache Parquet using user defined encodings (via WASM). Favorite image shows the ease of integrating into @apachedatafusion.bsky.social gienieczko.com/anyblox-paper

I am speaking at the #ApacheIceberg NYC Meetup on July 10th about Variant in #ApacheParquet which enable more efficient of processing semi structured data such as that found in JSON. lu.ma/95a5qys1

Quite a list of contributors already to the Rust Apache Parquet implementation of Variant (support for semi structured data). I was making some slides to explain what Variant is and made up a list I wanted to share. The feature will be amazing github.com/apache/arrow...

I publicly apologize for snapping today with @andypavlo.bsky.social @pateljm.bsky.social. "You need to have a push based scheduler to do ..." TUM / DuckDB(CWI) created group-think in Databases where push schedulers are required, ClickHouse, Spark, DataFusion, etc not withstanding 🤦

New blog post about cooperative scheduling using tokio and Rust async, and how cancellation works in @apachedatafusion.bsky.social datafusion.apache.org/blog/2025/06...

Two new (reposted) blogs about Optimizing SQL and DataFrames: datafusion.apache.org/blog/2025/06... datafusion.apache.org/blog/2025/06...

This is so cool -- an example of embedding a special index (a DistinctValues index no less) inside a Paruqet file: github.com/apache/dataf... (coming in DataFusion 49.0.0) @apachedatafusion.bsky.social