Smallpond by DeepSeek!
I'm a bit late, but grateful that DeepSeek made their (distributed) data processing framework OSS.
Take-away?
<= 10TB? keep using Polars/DuckDB
> 10TB? think about the complexity added
But! I believe the future holds simpler tooling.
A small blog below!
9 months ago