Appraising ML prediction models @maartenvsmeden.bsky.social 1) Share model and code 2) Predictors and language 3) Class imbalance & calibration 4) Model validation sample size 5) Is the model clinically useful in practice? www.jclinepi.com/article/S089...

Fantastic new paper casting doubt on explainability of explainable AI. To explain complex machine learning algorithms you need reproducibility of the explanation at a minimum academic.oup.com/ehjdh/articl... #machinelearning #Statistics #StatsSky @maartenvsmeden.bsky.social

Happy to see this in print! doi 10.1146/annurev-statistics-042324-123749 @maartenvsmeden.bsky.social @laurewynants.bsky.social @vanamsterdam.bsky.social and Ewout Steyerberg

I have got the data and I figured out exactly how to do my cluster analysis all I need is a relevant question that my cluster analysis is going to answer

What if you combine open datasets with AI? Apparently, a 3 fold increase in low quality research papers, mass-produced by paper mills. Interesting study in @jclinepi.bsky.social #academicsky #episky #medsky #Skystats Thanks to @maartenvsmeden.bsky.social for initially posting this on Linkedin!

Our guidance regarding performance measures for medical AI models is finally out! - Stop bashing AUROC, although it does not settle things - Calibration and clinical utility are key - Show risk distributions - Classification statistics (e.g. F1) are improper www.thelancet.com/journals/lan...

NEW PAPER The use of explainable AI in healthcare evaluated using the well known Explain, Predict and Describe taxonomy by Galit Shmueli link.springer.com/article/10.1...

🧐

this is one of my favourite observations about sample size calculations. (afaik first articulated by Miettinen in 1985)

For some research studies the optimal sample size should be estimated at 0

“Data available upon reasonable request” is academic language for you can get my data OVER MY DEAD BODY

Manuscript_Final_Version_actualFINALcopy_version9b_USETHISONE.docx

Kind reminder: data driven variable selection (e.g. forward/stepwise/univariable screening) makes things *worse* for most analytical goals

NEW FULLY FUNDED PHD POSITION Looking for a motivated PhD candidate to join our team. Together with Danya Muilwijk, Jeffrey Beekman and I, you will explore opportunities and limitations of AI in the context of organoids For more info and for applying 👉 www.careersatumcutrecht.com/vacancies/sc...

Interpretable "AI" is just a distraction from safe and useful "AI"

No.

I wonder who those people are who come here dying to know what GenAI has done with some prompt you put in

If you think AI is cool, wait until you learn about regression analysis

NEW PREPRINT Explainable AI refers to an extremely popular group of approaches that aim to open "black box" AI models. But what can we see when we open the black AI box? We use Galit Shmueli's framework (to describe, predict or explain) to evaluate arxiv.org/abs/2508.05753

The healthcare literature is filled with "risk factors". This word combination makes research findings sound important by implying causality, while avoiding direct claims of having identified causal associations that are easily critiqued.

When forced to make a choice, my choice will be logistic regression model over linear probability model 103% of the time

Post just up: Is multiple imputation making up information? tldr: no. Includes a cheeky simulation study to demonstrate the point. open.substack.com/pub/tpmorris...

You can have all the omni-omics data in the world and the bestest algorithms, but eventually a predicted probability is produced & it should be evaluated using well-established methods, and correctly implemented in the context of medical decision making. statsepi.substack.com/i/140315566/...

Depending which methods guru you ask every analytical task is “essentially” a missing data problem, a causal inference problem, a Bayesian problem, a regression problem or a machine learning problem

* New preprint led by Joao Matos & @gscollins.bsky.social "Critical Appraisal of Fairness Metrics in Clinical Predictive AI" - Important, rapidly growing area - But confusion exists - 62 fairness metrics identified so far - Better standards & metrics needed for healthcare arxiv.org/abs/2506.17035

Surprisingly common thing: comparisons of prediction models developed using, say, Logistic Regression, Random Forest and XGBoost with conclusion XGBoost is "good" because it yields slightly higher AUC than LR or RF using the same data Fact that "better" doesn't always mean "good" seems lost

Published: the paper 'On the uses and abuses of Regression Models: a Call for Reform of Statistical Practice and Teaching' by John Carlin and Margarita Moreno-Betancur in the latest issue of Statistics in Medicine onlinelibrary.wiley.com/doi/10.1002/... (1/8)

What is common knowledge in your field, but shocks outsiders? Validated does not mean it works as intended. It means someone has evaluated it (and may have concluded it doesn’t work at all)

**New Lancet DH paper** "Importance of sample size on the quality & utility of AI-based prediction models for healthcare" - for broad audience - explains why inadequate SS harms #AI model training, evaluation & performance - pushback to claims SS irrelevant to AI research 👇 tinyurl.com/yrje52fn

People always ask me, “how do I know my manuscript is done?” There’s only one way, my friends. If your file name looks something like this: Manuscript - Final Draft 3.7 FINAL FINAL - FINAL (5).docx Then, and only then, is it time.

Re-proposing the Occam's taser: an automatic electric shock for anyone riding the AI hype train making their models unnecessarily complex

Rule of thumb: If your model requires data to look like this (balanced after SMOTE), then maybe you want to use a different model

We should really ban all use of AI from all our education because the use of AI will make our students dumb, and banning innovation has worked really well before

As scientists it is difficult to stay in love with the questions once you fall in love with the answers

So glad they let us use this turn-of-phrase @maartenvsmeden.bsky.social doi.org/10.1016/j.jc...

So.... about using large language models (e.g. chatGPT) for writing motivation letters for a job I get it! And honestly, use all the technology you need to write the best letter you can But after reading dozens of letters with almost EXACTLY the same intro paragraph I do get a bit tired of it

Only 5 days left to apply for our 3 PhD positions at the @umcutrecht.bsky.social 🎓 🏃🏃‍♀️

Code or it did not happen

‘Wow, this treatment was no better than placebo – because the placebo was so effective!’ It’s such a shame the term ‘placebo effect’ is so widely known and used. There are fundamental challenges with identifying it and most of the claims don’t even try. It’s just change-from-baseline every time.

Before citing a paper the least you can do is thoroughly read… 1st year student: the entire literature on the subject Grad student: the paper you cite Post-grad: title and abstract and scan the rest of the paper Professor: the title. PLEASE read the title

recommendations on which prediction model performance measures to report and which not too by @benvancalster.bsky.social more discussion here… arxiv.org/abs/2412.10288 #memtab

next up is @joieensor.bsky.social talking about his work on software implementation for sample size calculations for targeting precise risk predictions - code available to do this available here github.com/JoieEnsor/pm... #memtab

Still some spots available in our summer school on all things causal inference, 7-11 July in Utrecht! Discounts for those working in universities and non-profits, and affordable accommodation offered by @utrechtuniversity.bsky.social summer school!

HIRING! We offer 3 fully funded PhD positions to work on research methodology with experts in a great medical research environment. More info: www.linkedin.com/feed/update/...

Shouldn't need saying, but clearly it does "Reporting guideline checklists are not quality evaluation forms: they are guidance for writing" --> onlinelibrary.wiley.com/doi/10.1002/... ...quality/risk of bias assessment is aided by transparent reporting though.

📢 #MSKBiostats is excited to welcome our next #Biostatistics Seminar Speaker: Alex Carriero, PhD Candidate from UMC Utrecht (Netherlands) 📅 Wed. 4/23/25 at 10:30 AM ET 📩 Email Michelle Dongel ([email protected]) for zoom link & to register! @alcarriero.bsky.social @maartenvsmeden.bsky.social

Periodic reminder....😀 #Transparency is one of the six core principles underpinning the WHO guidance on #ethics and #governance of #ArtificialIntelligence for health --> who.int/publications... Important to adhere to the TRIPOD+AI reporting standards🙏 --> bmj.com/content/385/... #trustworthyAI

NEW PAPER: Updated CONSORT-2025 for reporting randomised trials is now available in the @bmj.com @jama.com , @thelancet.bsky.social, @plos.org and @naturemedicine.bsky.social —> www.bmj.com/content/389/... #openscience #transparency #medsky #statssky #episky

Industry that promises "explainability" to patients of their AI or other complex analytical models should be held against the highest standards which they will not be able to meet