The PyData Amsterdam 2025 keynote “Minus Three Tier: Data Architecture Turned Upside Down” by @hannes.muehleisen.org is out now. www.youtube.com/watch?v=DxwD...

New database leaderboard from Yellowbrick ranks the quality of DBMS optimizer estimates and plans. They only evaluate TPC-H for now and report results for Postgres + DuckDB + MSSQL: sql-arena.com/components/p... Repo: github.com/sql-arena/db... LinkedIn Group: www.linkedin.com/groups/15775...

Today's Future Data Systems Seminar Speaker: Ian Cook (@ian.columnar.tech) will present @columnar.tech's work on Apache Arrow's database connectivity API (ADBC). ADBC is available in modern DBMSs. Zoom talk open to public at 4:30pm ET. YouTube video available after: db.cs.cmu.edu/events/futur...

Today's Future Data Systems Seminar Speaker: Will Manning (@willmanning.com) will present @spiraldb.com's Vortex file format. Vortex is now a @linuxfoundation.org project. Zoom talk open to public at 4:30pm ET. YouTube video available after: db.cs.cmu.edu/events/futur...

Processing 100Tb of CSV files on a single machine is insane, little over 1hr per query, even if on a powerful AWS instance. Question heavily the need for complex systems when this is what’s possible now. Can’t wait for full write-up. Incredible work. duckdb.org/2025/10/09/b...

Taking the DuckDb hoodie on a trip. Not exactly Amsterdam but I’ve heard they like columnar databases here too.

Congrats to DuckDB team on LTS release w/ many great improvements! Hidden among them you can now use Hive filtering with read_blob, and SHOW TABLES FROM specific db w/o USE.

📈 DuckDB 1.4.0 is out! This is our first LTS release which comes with *one year of community support*. It also supports database encryption, the MERGE SQL statement and Iceberg writes. For more details, read the announcement blog post at duckdb.org/2025/09/16/a...

I tried loading eBird data (1.5B rows CSV ZIP) using DuckDB for fun, inspired by a Clickhouse blog post and a bit of curiosity. Both did well, DuckDB slightly faster querying and Parquet ingest, Clickhouse w/ native zip support, optimized for ingest and multitenancy. xevix.medium.com/ebird-in-duc...

Vol:18 No:8 → Saving Private Hash Join 👥 Authors: Laurens Kuiper, Paul Gross, Peter Boncz, Hannes Mühleisen 📄 PDF: https://www.vldb.org/pvldb/vol18/p2748-kuiper.pdf

Is there too much duplicated effort in data tools? I sometimes wonder about this. xevix.medium.com/data-tool-co...

Compiling DuckDB on Windows 11 (ARM) using UTM VM on macOS to debug Windows compile issues. It's a shame msvc doesn't exist outside of Windows, mingw/clang don't work the same and cross-compiling is tricky. Compiling takes 5-10 mins (instead of 1-2 mins native), but it works 🎉!

Stretching DuckDB w/ Common Crawl, ~1.7B rows, ~300 parquet files. ~2-3s for single-column aggregations, ~2-3 mins to SUMMARIZE the data, peaking at ~12-14GB memory usage. Not exactly real-time, but the fact you can do this on a laptop with no server setups or Spark pipelines is still amazing.

Neat little hack to get Hive partition list in DuckDB, useful for an overview. Might be neat to have built-in. gist.github.com/xevix/04f33d...

Added an Automator quick action to run sqlfluff for formatting SQL in browser fields, used here in the DuckDB UI. Only needs sqlfluff, optionally configure rules. Would be cool to get built-in one day, but works for now.

Apache Drill allowed storing metadata in an RDBMS, Iceberg scaling data, Arrow scaling columnar memory, Parquet columnar storage, Spark distributed compute, DuckDB single-node compute. DuckLake scales metadata and storage w/ compute on single node. Motherduck distributes compute.

Shots fired by @firebolthq.bsky.social with their new on-prem executable (www.firebolt.io/blog/introdu...). They have dethroned the Umbra system by The Germans™ at ‪@tum.de in the ClickBench rankings: benchmark.clickhouse.com

Vibe coding NOAA GHCN weather visualization from scratch w/ Claude Code and DuckDB MCP. There's Evidence and other vis tools but I don't want a pre-cached set of data, I want it to query live. Cool that this can be put together w/o writing my own HTML/CSS, as a web backend dev 😅

Sped up Hive-partitioned query in DuckDB by directly adding Hive keys to the filepath rather than filtering in WHERE. 2-level Hive partition w/ 300 level-1, 100 level-2 partitions. 300ms -> 20ms filtering a single level-1 partition which is fantastic.

Good in-depth demonstration of the power of R-tree and H3 with benchmarks. www.architecture-performance.fr/ap_blog/spat...

Veo 3 to generate a duck writing SQL. Bias is still an issue, but pretty amusing.

Been trying to use Cursor+Gemini Pro 2.5 in agent mode to add a new feature. It's gone into a lot of loops, and then fell into full-on desperation 😂. Laughing pretty hard at its attempts. I think software engineering is safe for a bit longer.

Tested Claude Desktop DuckDB MCP on 275 years of weather parquet data on my local filesystem and asking for analysis, and it concluded human-caused global warming unprompted 😶‍🌫️. Cool that it could tell when to check filesystem and when to use DuckDB for SQL querying.

New text to SQL benchmark from BirdSQL folks, with complex BI tasks etc. Latest models already doing very well at ~40% success. Mindblowing how many hours of work will be saved once T2S is widespread. livesqlbench.ai

What does DuckLake enable that we can't do today? Basic Datalake without Snowflake, without Iceberg's issues. Extreme scalability not needed for most people, but organized cloud-backed SQL querying is useful for a lot of people. Multiuser via e.g. Postgres and S3 is a low barrier to entry in 2025.

Today we're launching DuckLake, an integrated data lake and catalog format powered by SQL. DuckLake unlocks next-generation data warehousing where compute is local, consistency central, and storage scales till infinity. ⁠ducklake is an open standard and we implemented it in the "ducklake" extension.

Major Parquet performance improvements, DICT_FSST compression, TRY, direct CLI querying, and tons more I have to try. Surprised and humbled to see mention of a small feature I added to swap DBs on the fly, hope folks find it useful. Thanks team for making it easy to contribute and helpful review!

DuckDB v1.3.0 “Ossivalis” is out and it packs a ton of new features, fixes and optimizations. See the announcement blog post: duckdb.org/2025/05/21/a...

Wanted to catch up on duckdb developments between 1.2 and 1.3 (upcoming), lots of great work it seems. Created a log with just commit messages and asked Claude to summarize. Not too bad, but anyone know a better way? git log --pretty=format:"%s" --since "2025-02-04" > commits.log

After many, many hours of research and development, it's finally here… A detailed explanation of what "HDR" actually means, the problems it presents, and three possible solutions— along with a technical preview of the HDR support coming to Halide Mark II. www.lux.camera/what-is-hdr/

I've been working on a DuckDB extension to autoattach new DB files either locally or on S3, so an upstream process can generate new files and this can pull them in. WIP, and might not have time to work on it for a bit, but sharing just in case. github.com/xevix/duckdb...

DuckDB got a local UI. Thanks to our friends at MotherDuck, you can now interact with your DuckDB database through an interactive notebook, running on localhost. Read the announcement blog post at duckdb.org/2025/03/12/d...

Interviewing people in 2025 is a wild time. I do hope this forces interviewers to not rely on leetcode and instead be forced to find ways to confuse gen AI while testing for real skills. www.cnbc.com/2025/03/09/g...

Fantastic weather in Silicon Valley lately. Getting very green.

This worked like a charm. Can install privoxy via Homebrew as well. Unfortunate that browsers are becoming more user-hostile, but at least there are solutions.

Made a simple DuckDB extension for pretty formatting SQL strings in Rust using sqruff. Still alpha so not ready for real-life use. Would love to have built-in formatting in the DuckDB CLI, but cool that an extension can get partway there. github.com/xevix/duckdb...

This is awesome. The frustration with other parquet implementations is palpable but completely understandable. Parquet is as close to a common standard for columnar data as it gets, but can’t take advantage of many things depending on who produced the file, which is unfortunate.

Most of what we call "tech" is about getting the right information into a database and trying to prevent the wrong person from reading or updating it.

Compiling DuckDB on Steamdeck from source took under 11 minutes with ninja. The fans were blaring but CPUs did not melt… I think 🤔 . Will make a short script of dependency installation later.

Federal workers are speaking out today about ‘devastating’ losses to the agency that gives us our weather forecasts, plus a worrying scoop on office leases canceled for the National Centers for Environmental Prediction and a radar development lab #NOAA www.theverge.com/news/622990/...

Are you working with geospatial data, or have you been wondering why we even use it? Then, this article might be for you. Introducing & showcasing its ubiquitous use. Exploring DuckDB's spatial capabilities, transforming complex geographic analysis into simple SQL queries. Hacking on Foursquare.

It's amazing how good Python tools are for GIS compared to Java these days. geopandas, xarray, pyproj, gdal, shapely, cartopy, folium, even indirectly with libs like scikit for raster processing. I find Java libs hard to use, like NetCDF, Geotools, GeoAPI etc. Too much boilerplate, needs more docs.

DuckDB on a Steam Deck? Why not. Anecdotal testing shows it’s about 3x slower than M1, 6x slower than M4, but still subsecond queries on millions of rows. The new shell install script worked like a charm too 🎉

BUSPH Center for Health Data Science has developed, www.FindLostData.org a platform that provides a streamlined interface for locating datasets previously hosted on government websites. #busph SPH's Chad Milando shows us how to use this valuable tool. You can use it yourself here ➡️ findlostdata.org

In today's video I test the SiFive HiFive Premier P550, the fastest RISC-V development board I've used. ESWIN made a special case for this Mini-DTX board (like Mini ITX but a slight bit wider), and the board already has tons of the hardware working in Ubuntu, Debian, & Fedora

Here's a simple demo of ATTACH OR REPLACE functionality in DuckDB (nightly). Useful if data updated upstream, need to attach new file/source, but want to keep queries the same. github.com/xevix/duckdb...

Excited about upcoming Parquet improvements in DuckDB. TPCH (sf=10) ~60M rows writing to Parquet V2 went from 4.2s->1.3s (on M4 Max), 1.74Gb->1.43Gb. This greatly benefits ETL, even for one-off experiments from upstream data. github.com/duckdb/duckd... github.com/duckdb/duckd...

FreSSH bugs undiscovered for years threaten OpenSSH security

SQL does have its flaws. It’s tempting to ignore expert opinions on this, but it’s similar to how C/C++ devs shrug off Rust. Yes, you can build amazing things in C++, but memory management is very tricky. Great ideas by Profs. Neumann and Leis. Like PRQL but more SQL-like. youtu.be/TBAf5l1RmcA?...

ollama mlx support coming to mac 🎉. This should speed things up nicely for local models github.com/ollama/ollam...