ClickHouse solved Advent of Code 2025 puzzles using single ClickHouse queries. No UDFs, no temp tables, no preprocessing. Just pure SQL doing things SQL probably shouldn't do. Impressive and slightly cursed in the best way.

If you're wondering where database tech is really headed in 2026, Andy Pavlo's annual review cuts through the noise. Sharding, Postgres evolution, and hot takes you won't find in vendor blogs.

MarkItDown (Microsoft) converts a variety file types to Markdown optimized for LLMs. Handles structure, not just text. Pairs well with Kreuzberg for extraction (supports 50+ file formats!). If you're building RAG systems, these belong in your stack.

jax-js compiles NumPy-style array code into WebAssembly and WebGPU kernels that run entirely client-side. No server, no dependencies, just JAX's programming model in your browser. This changes what's possible for interactive ML demos.

Jan Van Haaren's 2025 soccer analytics review is a solid reference if you're doing applied ML in sports or want to see how domain experts handle sequential data. Covers spatio-temporal models, graph methods, Bayesian forecasting, and tracking data metrics. janvanhaaren.be/posts/soccer...

ML progress isn't driven by elegant theory. It's benchmarks, leaderboards, and engineering culture. In this post, Ben Recht explores why empirical testing beats clean math in practice and why that tension defines the field.

The paradox: show a complex graph and execs check their phones. Show a simple one and they demand endless breakdowns. The solution isn't more or less data. It's recentering discussions on the actual decision at hand. methodmatters.github.io/true-stories...

Your browser can run Python (Pyodide), execute OCR on PDFs, crop videos, and call LLM APIs—all without uploading anything to a server. The localStorage + CORS pattern makes surprisingly powerful tools possible with zero backend infrastructure.

Tired of charts that hide the story? Density plots + percentile intervals reveal what averages can't: full variability, extremes, historical context. Nine examples comparing Spain's temperature data show how geometry choice changes what readers understand.

Fisher arbitrarily chose p<0.05 a century ago and we've just... kept it. The problem: calling it "arbitrary" only works if you can suggest something less arbitrary. No one has, so here we are circling p-values close to 0.05 like it means something. vilgot-huhn.github.io/mywebsite/po...

Haskell for data science? dataHaskell adds dataframes, NSE-style column operations, and compiler optimizations that turn chained operations into single-pass computations. Immutability + strong types + functional composition might be the combo we've been missing. jcarroll.com.au/2025/12/05/h...

Learning SQL is like learning a foreign language: you need to read more variations than you'll actually write. Learn disciplined canonical syntax for your own queries, but understand the messy dialects others use.

Side projects still open more doors than traditional applications in data roles. The trick: build small, interesting things that signal skills and make your work discoverable. Especially relevant given the current job market.

Most time-to-event metrics are broken. Amazon thought customer support wait times were under 1 min. Bezos called in: 10+ minutes. Why? They only measured customers who stayed on hold long enough to be served. The ones who hung up? Never counted. www.counting-stuff.com/why-you-are-...

Hot take: Your analysts doing "some basic data engineering" is killing your analytics function. The MTA hired 5 dedicated data engineers and it unlocked everything else. Stop asking data scientists to maintain pipelines. www.mta.info/article/less...

Sometimes the best polars pattern is knowing when to exit the DataFrame. partitionby() splits data into a dict of frames, letting you process with list comprehensions. Cleaner than forcing everything through mapgroups() when further wrangling isn't needed.

Most devs treat AI coding agents like infinite context machines. Reality: a 200k token window fills fast. The /compact feature is a trap. Better approach: /clear + document state in markdown, then resume. Treat context like disk space. You need a cleanup strategy.

Most modern dimensionality reduction (t-SNE, UMAP, Isomap) shares a pattern: represent data as a graph capturing local similarity, then embed to preserve that structure. It's graphs all the way down.

I curated some readings for class on "data tensions" and the list felt worth sharing. Come on a tour of datasets, books, the web, and AI with me... We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet. 1/n

Everyone's rushing to pgvector for "simple" vector search in Postgres. This reality check shows what actually happens at scale: indexing nightmares and performance walls. Simple isn't always sustainable in production.

When healthcare becomes algorithmic, what gets optimized out? This Guardian essay asks the hard question about AI spreading through diagnostics and therapy: are we trading care quality for efficiency without realizing the cost?

Thinking Machines Lab solved a problem everyone accepted as unsolvable: LLM nondeterminism at temperature 0. Same prompt, same model, 1000 runs → 80 different outputs. With batch-invariant kernels? Bitwise identical every time. Open sourced. www.distributedthoughts.org/will-i-make-...

Most marketplaces have SKUs. Etsy has 100M+ unique items with no standard attributes. How do you build filters when one listing is a "porcelain sculpture that looks like a t-shirt" and dimensions live in random photo text? www.etsy.com/codeascraft/...

GeoUtil converts between GeoJSON, TopoJSON, Shapefile, KML, WKT, and CSV without touching a server. TopoJSON compression alone cuts file sizes 80%+ while preserving topology. All free, all browser-based. geoutil.com

Debugging constraint problems is backwards: remove constraints until something works, then figure out what broke. No stack traces, just an "unsatisfiable." Forces you to think differently about what you're actually asking the system to solve. www.righto.com/2025/10/solv...

Most interpretable models sacrifice accuracy. Most accurate models are black boxes. TRUST breaks this trade-off by combining parametric and non-parametric approaches and offers full prediction explanations without losing performance. Published at PRICAI 2025.

Data products shouldn't live forever by default. Netflix treats outdated metrics like deprecated software and actively sunset them. The cost of maintaining zombie datasets? Lost trust and accumulated technical debt that blocks innovation.

What will you be using this #30DayMapChallenge? Cadence is offering its users £2,500 worth of prizes this #30daymapchallenge. And every user (new or existing) gets a free Professional upgrade for November! Learn more: cadence.cityscience.com/blog/30-day-...

Why does neural network training almost never fail? Pure combinatorics. A 6K parameter network contains 10^1089 possible sparse subnetworks. That's 10^900 solutions per atom in the universe. We're not smart, we're just brute forcing.

The best part about #30DayMapChallenge is that it's tool-agnostic. Whether you're using QGIS, Python's geopandas, R's rayshader, or even Blender for 3D visualizations, the focus is on creativity over tech stack. No programming required.

Stop fitting separate binomial models to compositional data! Your predictions that 130% of respondents chose option A reveal fundamental model misspecification. Dirichlet regression with Gaussian processes respects constraints.

AI systems are now generating, testing, and validating their own hypotheses. DeepMind's Co-Scientist and Stanford's Virtual Lab represent something new: AI as actual scientific collaborator, not just a fancy search engine.

Black and white, hand-drawn data viz countering visual noise at one of NYC's busiest hubs. Smart choice. Sometimes the most effective data art isn't about adding more complexity but finding clarity in the chaos through intentional restraint.

The metrics that matter most are the hardest to measure. Bookings take weeks to materialize, but clicks happen instantly. The trap: optimizing for clicks can actually decrease bookings. Correlation isn't causation, especially in A/B tests.

Psychology's dirty secret: "data available upon request" usually means data not available at all. Researchers found systematic patterns in why data disappears over time. Open-washing is real and it's undermining reproducibility.

Today my @nytimes.com colleagues and I are launching a new series called Lost Science. We interview US scientists who can no longer discover something new about our world, thanks to this year‘s cuts. Here is my first interview with a scientist who studied bees and fires. Gift link: nyti.ms/3IWXbiE

Parquet is showing its age. CMU researchers built F3 - a columnar format that embeds WebAssembly decoders directly in files. Universal compatibility without the usual compatibility hell. Smart approach for modern ML workloads. db.cs.cmu.edu/papers/2025/...

Mathematicians feared nuclear winter would freeze Earth, but it turns out CO2 might do it instead. The math behind climate tipping points is fascinating and terrifyingly unpredictable. Sometimes the thing you're not watching is the real threat.

Data dictionary template: osf.io/ynqcu Project summary template: osf.io/q6g8d Dataset level README template: osf.io/tk4cb

"Silicon samples" - using LLMs to generate fake survey responses instead of recruiting humans. Sounds efficient until you realize small model tweaks completely flip your results. Shortcuts in research usually aren't.

Your ggplot2 charts work fine, but are they memorable? Real color engineering: brightness first (strongest differentiator), then hue, finally saturation. Most people get this backwards and wonder why their viz falls flat. www.chartography.net/p/color-engi...

Tuesday Picks!...

We're going to get back into sharing useful posts for the week. Here's this week's shortlist...

Technical writing is hard bcs "writing is thinking" but we often should tell our story not in the order we worked. Solution? I wrote a quick post on how @quarto.org 's embed shortcodes can reframe technical writing as reproducible evidence curation www.emilyriederer.com/post/quarto-... 🧵 (1/n)

We've released 4 new chapters of Applied Machine Learning for Tabular Data. Includes: Bayesian optimization, feature selection, model comparisons, classification metrics, calibration, #rstats computing sections, and more blog.aml4td.org/posts/2025-0...

🧪 If you had your NSF grant terminated, I'd *highly* recommend going to this Friday's webinar. Friday May 9 at 2-3 pm ET Will talk about both the appeals process, in addition to allowable closeout costs. Share widely. Register here: us02web.zoom.us/webinar/regi...

A huge barrier to learning RL has always been the technical setup. This new browser-based RL Playground removes all the friction. You can experiment with Q-Learning, SARSA, and more with just a few clicks. Open source too if you want to tinker. awjuliani.medium.com/the-beginner...

The 2025 AI Index shows a field leveling up fast: small models now rival giants, costs are plummeting, and China is closing the performance gap. AI use in business & healthcare is exploding, but so are harms. Here's a data-packed snapshot of the current state of AI: hai.stanford.edu/news/ai-inde...

Ladies, gentlemen, this outstanding piece of #climate information is out. Welcome to the European State of the Climate 2024 #ESOTC2024 Explore it, bookmark it, come back to it and then come back again, because there’s more in it than you can imagine. climate.copernicus.eu/esotc/2024

This was actually my longest podcast ever at over 70 minutes. Not sure I could have made it any shorter because nerding out on databases with Andy Pavlo was too fun.