A recurring conversation: - I asked AI [an LLM] to do a complicated thing that I couldn't have done myself and don't understand, and it did it - how cool is that?! - How did you work out whether it got the right answer? - 🤷‍♂️ - 😢

"Federal research funding has produced exceptional returns precisely because decisions about what to fund have rested on merit and independent expert judgment, rather than political alignment." @webmz.bsky.social comments on OMB's proposal rule for @cos.io www.cos.io/blog/what-om...

The emerging disenchantment with retrospective meta-analysis www.nature.com/articles/s41... Retrospective meta analysis lacks prior unobservability of data, hence all problems known from empirical research apply. Multilab preregistered studies are seen as best alternative.

New publication by Jasmijn Krujit, @bnbakker.bsky.social et al. in @cogitatiomac.bsky.social: Only a minority of studies in Communication transparently share research materials or data. Way to go ... #openscience www.cogitatiopress.com/mediaandcomm...

AUTOMATING COMPUTATIONAL REPRODUCIBILITY My colleague @philipjakobbln.bsky.social and I are currently engaged in a research project where we reproduce scientific results en masse. To that end, we built rigor.me, a platform for automatically reproducing papers using agents 🧵

New findings from Giving in the Netherlands: in 2024, the Netherlands donated € 5.49 billion. That's $ 6.38 billion or £ 4.77 billion. An increase from 2022, but in terms of GDP a decline from 0.58% to 0.54%. 70% of households give, and 45% of the population volunteers. osf.io/8fvbc/files/...

Our metacheck package now has an online manual! In 29 short chapters we explain how to set up metacheck (running all modules 100% locally if you want), and explain the 18 available modules. www.scienceverse.org/metacheck_bo... We are still in beta, and would love to get feedback!

📢 Call for Papers! We are partnering with the Journal of Economic Surveys on a special issue: Reproducibility and Meta-Science in Economics. Submission deadline: 1 March 2027. Details in thread 🧵

What happens when researchers stop competing for funding? Twenty Dutch metascientists replaced competition with collaboration for NWO grant applications, proving collective approaches can outperform individual ones. @sajedehra.bsky.social @lakens.bsky.social www.tue.nl/en/news-and-...

Very pleased to have been on the leadership team for this paper! LLMs are already being used all over behavioural science. But it is often pretty hard to work out exactly what has been done, and therefore how much confidence to place in the results. www.nature.com/articles/s41...

My PhD Sajedeh Rasti has published a World View piece in Nature today: "Don’t compete, collaborate: why collective funding applications are the future" www.nature.com/articles/d41... She discusses how she successfully brought together a team of meta-scientists to collaboratively apply for funding.

Only 9.9% of top Sociology articles from 2019-2024 provided replication packages. And ≈ half of those with replication packages could not be verified due to missing or incomplete materials. www.econstor.eu/bitstream/10...

1/ Can AI help researchers check whether published social science results actually reproduce? In our new PNAS paper, we tested this directly in the AI Replication Games: 288 researchers, 103 teams, and real replication packages from quantitative social science.

Should automated tools for peer review come with a "nutritional label" for validity?🥪 And if so, what should be on it? Join my workshop on June 5th @ the META/e Conference on Transparency, Technology and AI in Peer Review in Eindhoven! Results will inform our FAIR validation framework. Details >>

Excellent news! Nature is expanding Registered Reports to all the fields in which they publish! A great result by all those proposing this strong solution to publication bias and selective reporting (and getting expert feedback before data collection)! www.nature.com/articles/d41...

Are you in the process of preregistering your study or thinking about performing it in future? The final workshop of this academic year at PMGS is exactly focusing on this. @denolmo.bsky.social will cover core principles of preregistration and common mistakes, and gives a hands-on guide for it.

A master student, Mink Veltman, is helping us to develop a miscitation detection module for Metacheck. If you want to help us to reduce miscitations, it would be great if you could take 5 minutes to fill out this survey: forms.cloud.microsoft/e/J7ugQw8mic Thanks in advance!

I also created a version of the 'guess the correlation' game. Students always enjoyed it, but the original website (with fun retro design) started to require a google account login. So I created my own version. Play it here: lakens.github.io/statistical_...

My article "Three-Sided Testing to Establish Practical Significance" with @jackfitzgerald.bsky.social is now published in AMPPS! journals.sagepub.com/doi/10.1177/... Three-sided testing is an improved version of TOST that lets you test for equivalence, superiority and inferiority simultaneously.

I’m working on the revised docs for the upcoming beta release of metacheck and am looking for advice on what kind of questions you’d have and what level of detail you’d want to know about data privacy and AI use.

The tools in Metacheck will be able to help with menial tasks like this. Read in PDFs and automatically check for matching entries in CrossRef. www.scienceverse.org/metacheck/

Reminder: If researchers find Cohen's d = 6, no they didn't. trustworthy.scientific.claims/posts/if-res...

Recently, I got quite frustrated listening to a large group of economists discussing preregistration and pre analysis plans. So, I've channelled that into an argument for why they should be preregistering their work whenever performing confirmatory research kdoroc.substack.com/p/arguing-wi...

We are inviting applications for a two-year postdoctoral position in a collaborative meta-science project on the effectiveness of data and code sharing policies in research-performing organizations. www.tue.nl/en/working-a...

Hope to see you at the Conference on Transparency, Technology and AI in Peer Review on June 5, 2026, which is organized in collaboration with the Center for Humans and Technology at Eindhoven University of Technology. Register at: meta-eindhoven.github.io/events/

The governing body for 14 Dutch universities (UNL) has published a "Vision on Publication Culture" that is so inspiring and forward thinking. Worth a read for those trying to changes publishing and research assessment practices. Take a read: www.universiteitenvannederland.nl/files/public...

Polling has become increasingly hard to do well, but THIS is not the solution. “Polling companies see an opportunity to use A.I. agents to simulate survey responses at a small fraction of the cost and time required for traditional polling.” Gift link: www.nytimes.com/2026/04/06/o...

New blog post (it's been a while): Make Meta-Analysis Great Again. This is a serious proposal. Please amplify to whoever might be interested, especially if they have money. steamtraen.blogspot.com/2026/04/make...

A project offering researchers up to 3,500 Swiss francs for finding errors in published academic papers is having trouble finding people to do the work. My latest for @science.org: www.science.org/content/arti... @erichehman.bsky.social, @malte.the100.ci, @ruben.the100.ci, @conjugateprior.org

A year ago, @sajedehra.bsky.social and I reached out to 20 Dutch metascientists and asked: Instead of competing for this upcoming grant, how about we collaborate, determine what we believe is a priority, and combine our expertise? Both grants were funded. We discuss our experience in a new preprint.

For replicability, we tested findings from 164 papers and successfully replicated 49% of them with the common statistical significance criterion. Original studies had an average effect size of r = 0.25, replication studies r = 0.10.

Every paper begins as a promise and ends as a compromise.

Should we use error checking tools like Statcheck and GRIM to automatically check for errors in studies in meta-analyses? I wrote a blog post with some insights from a hackathon where we scrutinized this idea. metaresearch.nl/blog/2026/3/...

…deviations are usually small and random. Most original and reproduced results are stable in sign and significance, and we find no bias towards exaggerated effect sizes in original research.

…reproducibility problems are widespread. Only one in three authors (35%) share code, and even when code is available, only every second main result (51%) is numerically reproducible.

Power After the Results are Known (PARKing) floggingpvalues.blog/2026/03/13/p...

Has transparency improved in clinical psychology? New cross sectional study from us (Bianca Kotoulas, Justine Blackwell & me). Adoption of transparent research practices has increased between 2012, 2018, & 2024, but aside from coi and funding disclosures, they remain uncommon osf.io/preprints/ps...

Tracking and mainstreaming replications in the social, cognitive, and behavioral sciences: https://doi.org/10.31222/osf.io/ad2w6_v1

When you collect data online, are the results from humans or AI? In a project led by Booth PhD student Grace Zhang, we estimate the prevalence of AI agents on commonly used survey platforms: osf.io/preprints/ps... 🧵

This post really put together the pieces in a way that floored me. Everything is about to change and we have to confront that reality causalinf.substack.com/p/claude-cod...

"the paper’s main outcome variable cannot possibly be constructed as described in the paper." Retraction of 2023 paper that did not use the reported variables. The replication report is astonishing www.nature.com/articles/s41... Pre-publication peer review remains undefeated in laundering bullshit

It’s finally out! Together with @embopress.org and @reviewcommons.org, we conducted a structured side-by-side comparison of human peer review and our AI scientific review (see thread 👇👇👇🔥).

Interesting predictions on what will happen "when “science as checklist” becomes policy"

1/ Sorry for double-posting from X. Sharing a new working paper for the Year of the Horce 🐎: "An AI-assisted workflow that scales reproducibility in empirical research" (bit.ly/repro-ai) w/ Leo Yang Yang

Deze collega's schrijven: "Zo berichte NRC over een studie die liet zien dat migratieonderzoekers met uiteenlopende ideologische opvattingen tot tegenstelde conclusies kwamen op basis van dezelfde dataset" Deze grafiek uit de studie laat zien dat dat n volstrekt onjuiste interpretatie is.

In a new paper with Eva-Maria Merz and Ting Li, we analyzed data from 5,967 respondents in the Wisconsin Longitudinal Study (WLS) to examine which characteristics of individuals and families create the association between educational attainment and engagement in prosocial behavior.

Wild how economists and political scientists worry so much about unbiased tests **in their papers** and yet basically ignore how their journals filter on significance. Given our noisy tests, the latter creates huge bias away from zero.

New paper, on a worrying trend in meta-science: the practice of anonymising datasets on, e.g., published articles. We argue that this is at odds with norms established in research synthesis, explore arguments for anonymisation, provide counterpoints, and demonstrate implications and epistemic costs.

My first paper is out in #SociologicalScience! With Jörg Stolz and Ruud Luijkx, we found robust evidence of ideological #bias in #secularization research: researchers' own religiosity is correlated with their probability of finding evidence of religious decline in their publications. Read more: 👇