Our paper on the effect of large language models on replicators' success in conducting reproducibility + robustness checks is now published at PNAS. Before advocating that we let AI completely take over the replication process, please read our paper! doi.org/10.1073/pnas...

Ok what's the funniest part? I'm personally a fan of the unhinged 'best of luck' email signoff, though a close second is starting the email off with 'Hi Doctor, Hey.'

Fresh preprint! Using Dutch administrative data, I show that businesses who respond to business surveys are typically unrepresentative, and that this unrepresentativeness is driven by which businesses are contacted for those surveys. 1/ doi.org/10.31235/osf...

My article "Three-Sided Testing to Establish Practical Significance" with @jackfitzgerald.bsky.social is now published in AMPPS! journals.sagepub.com/doi/10.1177/... Three-sided testing is an improved version of TOST that lets you test for equivalence, superiority and inferiority simultaneously.

Happy to see my paper with @isager.bsky.social on three-sided testing published at Advances in Methods and Practices in Psychological Science! If you're interested in practical significance testing, give it a read! doi.org/10.1177/2515...

Given my stance on log-like specifications, I was surprised to learn that there's a 'news' article on my paper in Nature Human Behaviour, claiming that it actually advocates for the use of IHS specifications. This is categorically untrue. 1/ t.co/ZGjAmHZhLv

New preprint! We reanalyze 46 papers that use log-like specifications (ln(Z+1), inverse hyperbolic sine etc). We find widespread non-robustness, and we show through theory + simulation how these models drive spurious significance. 1/ doi.org/10.31222/osf...

We're thrilled to open registration for the Utrecht Replication Games. The event will be at the at the University of Utrecht on June 4th. Psych, public health, pol sci and econ studies will be reproduced! Register here: www.surveymonkey.ca/r/Replicatio...

When your spam targets won't submit so you *demand submission* Wishing y'all luck today

Had a wonderful time organizing the scientific side of the CBS Replication Games! Thank you to the replicators for your hard work!

My Matters Arising concerning a paper on the legal determinants of terrorism is now out in @nathumbehav.nature.com. The original paper is now retracted. To learn why, read on for a story of irregularities, imputations, and impossible values. 1/x doi.org/10.1038/s41562-025-02347-7

Pleased to announce that my paper on hypothetical bias in experiments is now published in Experimental Economics! If you're interested in (how we figure out) whether real stakes and incentives matter in experiments, give it a read! 🔗: doi.org/10.1017/eec....

Had a great time presenting my job market paper at the Lindau Nobel Meeting in Economic Sciences! 🔗 : osf.io/d7sqr_v1/ #LINOecon #EconSky

Registration closes July 4! Come join us in-person or online in Amsterdam!

Dutch academics, please share - replication games are coming to Vrije Universiteit Amsterdam on July 19! If you live in the Netherlands, or want a good reason to visit, come take a day to network, hone your skills, and potentially coauthor a paper. Registration: www.surveymonkey.ca/r/Replicatio...

#GDRI_rep Update 7: Retraction! Our comment of "Parent–teacher meetings and student outcomes: Evidence from a developing country" has led to the paper being retracted. Our comment is accepted as is. Short 🧵

This will likely be a good initiative, conditional on it not substituting for the journal publishing full comments. Publication is an important incentive for reproducibility/robustness analyses and substantive empirical critiques almost always take more than one page.

#GDRI_rep Update 5: We have a new report. We reproduced the paper entitled "Partisan Effects of Information Campaigns in Competitive Authoritarian Elections: Evidence from Bangladesh" by Ahmed, Hodler and Islam published at the Economic Journal. See below for links to report and authors' responses.

What academic journal should I start? Wrong answers only

#GDRI_rep Update 4: We have a new report. We reproduced the paper entitled "Parent–teacher meetings and student outcomes" by Islam @ European Economic Review. See below for links to report and author's response.

Thank you @carlislerainey.bsky.social for boosting my job market paper! I’’m a fan of your prior work on this topic. If you want to learn more, check out the paper and my BIBAP seminar on it tomorrow, 3/12 7 AM CET/5 PM Sydney time. www.unsw.edu.au/business/beh...

#5 Here's another recent critique of close elections RDDs. It's unclear whether there is actually good evidence against potentially fatal precise manipulation around the cutoff in these designs.

I'll be waking up early (7 AM CET) on Tuesday, March 12 to present my job market paper at 5 PM Sydney time! If you're awake too, stop by to hear me talk about equivalence testing, replication-based methods research, and the robustness of null results in economics!

Relatedly, I also develop new equivalence testing methods for testing running variable (RV) manipulation in RDD. In 36 RDD publications, I find >44% of RV density discontinuities at treatment cutoffs can't be significantly bounded beneath 50% upward jumps. 🧵: bsky.app/profile/jack...

#GDRI_rep Update 2: I4R sent a report to the original authors for the PLOS One article "“Food insecurity and mental health of women during COVID-19: Evidence from a developing country". All authors have now emailed PLOS One to withdraw their names. osf.io/7tzek/

#GDRI_rep Update 1: I4R are now reproducing all published papers that use data from GDRI, or are closely related in other ways. Here is a first update on our work. 🧵

Our proposal for post-publication review: “our system would add only 1 percent to the total reviewing effort, while providing important perspectives on papers representing more than one-quarter of the citations received by these influential journals.” statmodeling.stat.columbia.edu/2025/02/26/pp/

The Need for Equivalence Testing in Economics

As a coauthor on the AEJ:AE report, it’s a lot. Oh my god, It’s a lot. In that paper, outcomes are inconsistently handled both in the code and in the field, the paper’s data is connected to a bunch of other experiments, and we find irregularities in the raw survey files.

I had a wonderful time presenting my job market paper at RWI Essen! Thank you @jrgptrs.bsky.social and Julian Rose for hosting and organizing!

New working paper! @i4replication.bsky.social does a good job detailing the experiment’s results, so for anyone who considers LLMs/AI as a (soon-to-be) solution to reproducibility in social science, let me walk you through the 🔥hell🔥 of trying to reproduce a paper on an "AI-led" team. 1/7

New paper for holiday reading! @isager.bsky.social and I provide an introduction to three-sided testing, a framework for testing estimates' practical significance. We offer a tutorial, Shiny app, + commands/code in #Rstats, #Jamovi, + #Stata. 1/9 osf.io/preprints/psyarxiv/8y925 #EconSky #PsychSky

New preprint: “Three-Sided Testing to Establish Practical Significance: A Tutorial” (osf.io/preprints/ps...) with @FitzgeraldJack_, proposing an upgraded version of the TOST equivalence test 🧵

A huge thanks to @sweiwang.bsky.social for highlighting my job market paper, The Need for Equivalence Testing in Economics!

In more recent news, I thoroughly enjoyed presenting The Need for Equivalence Testing in Economics at the Netherlands Reproducibility Network Symposium and Platform for Young Meta-Scientists Symposium, with great discussion from Tsz Keung Wong!

Two weeks ago, I had a wonderful time presenting The Need for Equivalence Testing in Economics at the Leibniz Open Science Day! (pic: @prashantgarg.bsky.social)

Moving from Florida to the Netherlands is a trip because for three months of the year, you can look to the sky and think ‘wow, what a beautiful sunset’ before realizing it’s 2:30 in the afternoon

Suppose I say “I regressed V1 on V2.” Which one do you assume is the dependent variable? 1: V1 2: V2 #EconSky #rstats #statistics

Super important resource for the field, give them a follow! #EconSky

I had an excellent time presenting this paper to the Behavioural Insights for Business and Policy Network at the University of New South Wales. A huge thanks to @impartialspectator.bsky.social for hosting!

Do real stakes/incentives matter in experiments? Recent studies say they don’t. My new paper shows that these studies’ results — and those of most hypothetical bias experiments — are uninformative when we care about treatment effects. 1/19 #EconSky #PsychSky #PoliSky 🔗: papers.tinbergen.nl/24070.pdf

Here is the starter pack for folks in science of science, meta science, STS, philosophy of science, history of science, economics of science, science policy, AI+science, and so on. Feel free to nominate and self-nominate! #SciSci #STS #MetaScience #SciPol #AI4Science #ScAISci go.bsky.app/2R1E4TD

A research highlight for #EconSky! I design equivalence tests for running variable (RV) manipulation in regression discontinuity (RDD), show that RV manipulation can't be ruled out in lots of published RDD research, and offer the lddtest command in Stata/R. 1/x 🔗: hdl.handle.net/10419/300277

PSA to Dutch academics: It looks like the protest has been cancelled at the last minute. nos.nl/artikel/2544...

Protests are better together! If you're headed to the demonstration tomorrow in Utrecht to protest government cuts to higher education and research, and you'd like to meet up, feel free to reach out!

@o.simardcasanova.net has curated an excellent starter pack of folks working on economics of science and replication! bsky.app/starter-pack...

A great conversation about my experiencing publishing a replication with PNAS and about replication work more generally. This is a podcast worth tuning into!