My 2025 highlights for AI research and code: ▪ Unpacking the AI scale narrative ▪ Tabular-learning research - TabICL: table foundation model - Retrieve merge predict: data lakes ▪ Better software - Skrub: machine learning with tables - Fundamentals in scikit-learn gael-varoquaux.info/science/2025...

Sacha Braun, David Holzm\"uller, Michael I. Jordan, Francis Bach Conditional Coverage Diagnostics for Conformal Prediction https://arxiv.org/abs/2512.11779

Let's kick off 2026 with a workshop on Survival Analysis and Foundation Models, co-organized w. Julie Alberge Linus Bleistein Clément Berenfeld Agathe Guilloux and Julie Josse on January 27th at PariSanté Campus ! Registrations and submissions are open!!! www.linusbleistein.com/ramh

Still using temperature scaling? With @dholzmueller.bsky.social, Michael I. Jordan and @bachfrancis.bsky.social we argue that with well designed regularization, more expressive models like matrix scaling can outperform simpler ones across calibration set sizes, data dimensions, and applications.

⚡ Release 0.6.2 is out ⚡ github.com/skrub-data/s...

Daniel Beaglehole, David Holzm\"uller, Adityanarayanan Radhakrishnan, Mikhail Belkin: xRFM: Accurate, scalable, and interpretable feature learning models for tabular data https://arxiv.org/abs/2508.10053 https://arxiv.org/pdf/2508.10053 https://arxiv.org/html/2508.10053

I got 3rd out of 691 in a tabular kaggle competition – with only neural networks! 🥉 My solution is short (48 LOC) and relatively general-purpose – I used skrub to preprocess string and date columns, and pytabkit to create an ensemble of RealMLP and TabM models. Link below👇

Excited to have co-contributed the SquashingScaler, which implements the robust numerical preprocessing from RealMLP!

👨‍🎓🧾✨#icml2025 Paper: TabICL, A Tabular Foundation Model for In-Context Learning on Large Data With Jingang Qu, @dholzmueller.bsky.social, and Marine Le Morvan TL;DR: a well-designed architecture and pretraining gives best tabular learner, and more scalable On top, it's 100% open source 1/9

🚨What is SOTA on tabular data, really? We are excited to announce 𝗧𝗮𝗯𝗔𝗿𝗲𝗻𝗮, a living benchmark for machine learning on IID tabular data with: 📊 an online leaderboard (submit!) 📑 carefully curated datasets 📈 strong tree-based, deep learning, and foundation models 🧵

Missed the school? We have uploaded recordings of most talks to our YouTube Channel www.youtube.com/@AutoML_org 🙌

📝 The skrub TextEncoder brings the power of HuggingFace language models to embed text features in tabular machine learning, for all those use cases that involve text-based columns.

🚨ICLR poster in 1.5 hours, presented by @danielmusekamp.bsky.social : Can active learning help to generate better datasets for neural PDE solvers? We introduce a new benchmark to find out! Featuring 6 PDEs, 6 AL methods, 3 architectures and many ablations - transferability, speed, etc.!

The Skrub TableReport is a lightweight tool that allows to get a rich overview of a table quickly and easily. ✅ Filter columns 🔎 Look at each column's distribution 📊 Get a high level view of the distributions through stats and plots, including correlated columns 🌐 Export the report as html

#ICLR2025 Marine Le Morvan presents "Imputation for prediction: beware of diminishing returns": poster Thu 24th arxiv.org/abs/2407.19804 Concludes 6 years of research on prediction with missing values: Imputation is useful but improvements are expensive, while better learners yield easier gains.

Excited to share the new monthly Table Representation Learning (TRL) Seminar under the ELLIS Amsterdam TRL research theme! To recur every 2nd Friday. Who: Marine Le Morvan, Inria (in-person) When: Friday 11 April 4-5pm (+drinks) Where: L3.36 Lab42 Science Park / Zoom trl-lab.github.io/trl-seminar/

We are excited to announce #FMSD: "1st Workshop on Foundation Models for Structured Data" has been accepted to #ICML 2025! Call for Papers: icml-structured-fm-workshop.github.io

Trying something new: A 🧵 on a topic I find many students struggle with: "why do their 📊 look more professional than my 📊?" It's *lots* of tiny decisions that aren't the defaults in many libraries, so let's break down 1 simple graph by @jburnmurdoch.bsky.social 🔗 www.ft.com/content/73a1...

🚀Continuing the spotlight series with the next @iclr-conf.bsky.social MLMP 2025 Oral presentation! 📝LOGLO-FNO: Efficient Learning of Local and Global Features in Fourier Neural Operators 📷 Join us on April 27 at #ICLR2025! #AI #ML #ICLR #AI4Science

Practitioners are often sceptical of academic tabular benchmarks, so I am elated to see that our RealMLP model outperformed boosted trees in two 2nd place Kaggle solutions, for a $10,000 forecasting challenge and a research competition on survival analysis.

A new tabular classification benchmark provides another independent evaluation of our RealMLP. RealMLP is the best classical DL model, although some other recent baselines are missing. TabPFN is better on small datasets and boosted trees on larger datasets, though.

Learning rate schedules seem mysterious? Why is the loss going down so fast during cooldown? Turns out that this behaviour can be described with a bound from *convex, nonsmooth* optimization. A short thread on our latest paper 🚞 arxiv.org/abs/2501.18965

Early stopping on validation loss? This leads to suboptimal calibration and refinement errors—but you can do better! With @dholzmueller.bsky.social, Michael I. Jordan, and @bachfrancis.bsky.social, we propose a method that integrates with any model and boosts classification performance across tasks.

The first independent evaluation of our RealMLP is here! On a recent 300-dataset benchmark with many baselines, RealMLP takes a shared first place overall. 🔥 Importantly, RealMLP is also relatively CPU-friendly, unlike other SOTA DL models (including TabPFNv2 and TabM). 🧵 1/

Join us on 27 Feb in Amsterdam for the ELLIS workshop on Representation Learning and Generative Models for Structured Data ✨ sites.google.com/view/rl-and-... Inspiring talks by @eisenjulian.bsky.social, @neuralnoise.com, Frank Hutter, Vaishali Pal, TBC. We welcome extended abstracts until 31 Jan!

My book is (at last) out, just in time for Christmas! A blog post to celebrate and present it: francisbach.com/my-book-is-o...

I'll present our paper in the afternoon poster session at 4:30pm - 7:30 pm in East Exhibit Hall A-C, poster 3304!

We wrote a benchmark paper with many practical insights on (the benefits of) active learning for training neural PDE solvers. 🚀 I was happy to be a co-advisor on this project - most of the credit goes to Daniel and Marimuthu.

Stable model scaling with width-independent dynamics? Thrilled to present 2 papers at #NeurIPS 🎉 that study width-scaling in Sharpness Aware Minimization (SAM) (Th 16:30, #2104) and in Mamba (Fr 11, #7110). Our scaling rules stabilize training and transfer optimal hyperparams across scales. 🧵 1/10

I'll be at #NeurIPS2024 next week to present this paper (Thu afternoon) as well as a workshop paper on active learning for neural PDE solvers. Let me know if you'd like to chat about tabular data, uncertainty, active learning, etc.!

Proud to announce our NeurIPS spotlight, which was in the works for over a year now :) We dig into why decomposing aleatoric and epistemic uncertainty is hard, and what this means for the future of uncertainty quantification. 📖 arxiv.org/abs/2402.19460 🧵1/10

If you have train+validation data, should you refit on the whole data with the stopping epoch found on the train-validation split? In the quoted paper, we did an experiment including 5-fold ensembles on a 5-fold cross-validation splits (bagging) and with refitting. (short 🧵)

I recently shared some of my reflections on how to use probabilistic classifiers for optimal decision-making under uncertainty at @pydataparis.bsky.social 2024. Here is the recording of the presentation: www.youtube.com/watch?v=-gYn...

One thing I learned from this project is that accuracy is a quite noisy metric. With small validation sets (~1K samples), hyperparameter opt. using AUROC instead of accuracy can yield better accuracy on the test set. We also did some experiments on metrics for early stopping. 🧵

PyTabKit 1.1 is out! - Includes TabM and provides a scikit-learn interface - some baseline NN parameter names are renamed (removed double-underscores) - other small changes, see the readme. github.com/dholzmueller...

WIP starterpack w researchers on Table Representation Learning (TRL): all things related to representation learning and generative models for e.g. tables, DBs, spreadsheets! I'll curate but DM/reply w handle+some info welcome! Also follow @trl-research.bsky.social for updates 🤗 go.bsky.app/4SNSMRj

For those who missed this post on the-network-that-is-not-to-be-named, I made public my "secrets" for writing a good CVPR paper (or any scientific paper). I've compiled these tips of many years. It's long but hopefully it helps people write better papers. perceiving-systems.blog/en/post/writ...

@bsky.app is the new cool. So is tabular learning? skrub is a library that eases preprocessing and feature engineering for tabular machine learning. skrub-data.org/stable/ Main features: - scikit-learn compatible - handles Pandas and Polars dataframes - works on heterogeneous types

Can deep learning finally compete with boosted trees on tabular data? 🌲 In our NeurIPS 2024 paper, we introduce RealMLP, a NN with improvements in all areas and meta-learned default parameters. Some insights about RealMLP and other models on large benchmarks (>200 datasets): 🧵

I created a starter pack of scientists in the European Laboratory for Learning and Intelligent Systems (ELLIS) 🇪🇺 Please ping me and I‘ll add you. go.bsky.app/Cihupkk

Hi 👋 I'm David Holzmüller, a postdoc at INRIA Paris. I work on supervised (deep) learning, uncertainty quantification, and active learning for tabular data (and some AI4Science). I like to post threads about my research. So please follow me if you like high-SNR "science Twitter" content 🙂

The video and slides of my talk are online (link is in the quoted tweet). 📽️ https://twitter.com/DHolzmueller/status/1831385303405281347

Tomorrow I'll be giving a talk on Generalization Theory of Linearized Neural Networks at the MML seminar: https://www.mis.mpg.de/events/event/generalization-theory-of-linearized-neural-networks The (virtual) talk will be live-streamed (5pm CEST = 11am EST = 8am PST) and a...

See you in 35 minutes at poster 218! https://twitter.com/DHolzmueller/status/1787410246895288689

I am at ICLR and would love to chat about tabular data, deep active learning, NN uncertainty, etc.! @ViktorZaverkin and I will be presenting a poster for our JMLR paper on deep batch active learning for regression on Tuesday, 10:45 - 12:45, Hall B,...

Here are two neat kernel theory facts: 📝 (1) Inverse kernel matrices can be studied through norms of minimum-norm interpolators (2) Equivalent kernels have equivalent kernel matrices (details below)

Happy new year in machine learning!

I'll present our work at NeurIPS@Paris on Wednesday in the first session (13:15 - 14:45). https://twitter.com/DHolzmueller/status/1717907414458966448

Can kernel methods and wide neural networks overfit benignly in fixed dimension? In our NeurIPS 2023 paper with @moritz_haa, @UlrikeLuxburg, and Ingo Steinwart, we prove: yes, but only with some non-standard modifications. 1/ 🧵

Last week, I successfully defended my PhD! 🥳🎉🥂 I want to thank everyone that supported me on this journey, especially Ingo Steinwart, @BachFrancis, @ViktorZaverkin, @SimTechStuttga2. Looking forward to a new postdoc adventure with @BachFrancis and @GaelVaroquaux!