Dani Solà (@dani-sola.com)

Incremental Jobs and Data Quality Are On a Collision Course - Part 1 - The Problem — Jack Vanlightly Big data isn’t dead; it’s just going incremental If you keep an eye on the data space ecosystem like I do, then you’ll be aware of the rise of DuckDB and its message that big data is dead. The idea comes from two industry papers (and associated data sets), one from the Redshift team (paper and dataset) and one from Snowflake (paper and dataset). Each paper analyzed the queries run on their platforms, and some surprising conclusions were drawn – one being that most queries were run over quite small data. The conclusion (of DuckDB) was that big data was dead, and you could use simpler query engines rather than a data warehouse. It’s far more nuanced than that, but data shows that most queries are run over smaller datasets. Why? https://jack-vanlightly.com/blog/2024/11/13/incremental-jobs-and-data-quality-are-on-a-collision-course-part-1-the-problem