Trying to reproduce results from old papers can take you to interesting places

International #ResearchIntegrity conference 16-18 November 2025 Come to hear @elisabethbik.bsky.social Ivan Oransky @jamesheathers.bsky.social @retractionwatch.com Lisa Bero @liammannix.bsky.social @jdwilko.bsky.social @jacksonwryan.com +many others, stay for Sydney in late Spring @sydney.edu.au 🧪

Nearly 3 months since I contacted both the authors and editor of @jama.com about failure to share data. Article states data and code is available, but the github repo is empty. Not private or partial, empty. Has never contained anything. No replies from anyone. jamanetwork.com/journals/jam...

For Monday crowd. INSPECT-SR, a tool to help you identify problematic (inc. fraudulent) RCTs, now available.

📚 INSPECT-SR: a tool for assessing trustworthiness of randomised controlled trials www.medrxiv.org/content/10.1...

Science is not always as good as it appears on print in a research paper. Ready to be scared by finding nasty truths. Try out INSPECT-SR for checking the trustworthiness of trial results when you do systematic reviews.

www.medrxiv.org/content/10.1... INSPECT-SR: A tool for assessing trustworthiness of randomised controlled trials.

One of the most common misunderstandings about the use and value of placebos in clinical RCTs, often made by both methodological experts and experienced trialists: (from @stephensenn.bsky.social in academic.oup.com/jrsssa/advan...)

Hoping INSPECT-SR will appear on medRxiV at some point today…

Last chance to participate - this will close on Sunday pm UK time. These results will be used to formulate guidance on demonstrating promise of an intervention when applying for funding for an RCT, pilot, or feasibility trial.

A letter from myself and @jdwilko.bsky.social that expands on this a bit more. Thanks to Fertility and Sterility for the opportunity. doi.org/10.1016/j.fe...

How to demonstrate ‘promise of the intervention’ ahead of an RCT of the intervention? If you’d like to participate in the final round of the Promise Delphi, even if you didn’t do Round 1, then get in touch!

Any OSF-heads have suggestions here? I noticed a typo in a public OSF document I had uploaded. So I deleted and uploaded the corrected version. But the new version (a Word document, same as the previous doc) won't display - 'Unexpected server response'. Any ideas?

Did you have a chance to catch @davidhosch.bsky.social, @neiloconnell.bsky.social, @jdwilko.bsky.social, @nadiasoliman.bsky.social, @lesleyuttley.bsky.social discuss “trustworthiness” in clinical trial data? Watch the recording here bit.ly/4luQVMS

Happening now!

Have you wanted to understand what is meant by “trustworthy” research? Attend this #webinar where @davidhosch.bsky.social, @neiloconnell.bsky.social, @jdwilko.bsky.social, @nadiasoliman.bsky.social, @lesleyuttley.bsky.social, and more outline how to produce reliable evidence bit.ly/44GBtGP

🔍 Systematic reviewers are the first line of defense for trustworthy science! 🛡️💡 Here’s why they must lead the charge for #ResearchIntegrity 👇 👉 www.cochrane.org/about-us/new... #EvidenceBased #MetaAnalysis #OpenScience 🧠📊

Nice piece on the tool that @cochrane.org are collaborating on - "Investigating Problematic Clinical Trials in Systematic Reviews (INSPECT-SR)" – to identify potentially problematic randomized controlled trials in systematic reviews Shout out to @jdwilko.bsky.social for all his work on this

Great to see increasing focus on the importance methodological rigour for the ethical conduct of patient focused research (thank you WMA Helsinki). Yet statisticians and other methods experts seem to often be left out of the conversation about solutions. statsepi.substack.com/p/everybodys...

Have you wanted to understand what is meant by “trustworthy” research? Attend this #webinar where @davidhosch.bsky.social, @neiloconnell.bsky.social, @jdwilko.bsky.social, @nadiasoliman.bsky.social, @lesleyuttley.bsky.social, and more outline how to produce reliable evidence bit.ly/44GBtGP

As soon as you say ‘it is unethical to randomly allocate people to a placebo’, you have shown you are not a serious commentator. You don’t understand the first thing about what trials involve.

What's that? The target trial emulation obtained the same answer as the RCT? Wow, I guess RCTs are generalisable after all! Perhaps we should perform them rather than using interventions on hundreds of thousands of people before we know whether or not they work, then doing a TTE?

0197eb9d-ef92-8d6e-7fbd-dcdffc914833.share.connect.posit.cloud You can tell me what you want (within reason) and I'll do it. Alternatively, you can make a PR on my repo github.com/Dpananos/chi...

I'm looking for a simple online calculator that will perform a chi-squared test and also a variety of common (not necessarily ideal) alternatives (e.g. Yates' correction, Fisher test - the more coverage the better). Ideally, it would present all of these results at once. Any suggestions?

This Delphi is live! Invitations are rolling out. Contact me if you have expertise in trials and would like to participate.

"Yes, nearly every statistical result in my paper contained an error, but significance changed for only one p-value" is not the strongest defence

Today’s challenge: explaining that long-run characteristics of estimators (e.g. A more precise than B) might not be evident in a particular instance (where B more precise than A).

Curious about trust in trials? FREE webinar: 21 July, 3pm UK, exploring what makes research trustworthy. Hosted by @iasp.bsky.social PRF & supported by the @jclinepi.bsky.social 👥 Speakers are leading trust and integrity researchers and journal editors. 💬 Plus live Q&A! shorturl.at/FEeMz

If you're interested in participating in this, register now (QR code in thread). Will launch next week, or, if I get myself in gear - later today! Expect the Delphi to take 20 mins.

Always risky giving a talk where you say 'seek input from a statistician' because afterwards 40% of the audience will email you for advice.

We applied a long list of trustworthiness checks to RCTs in 50 Cochrane Reviews with the goal of learning a bit more about the checks. A few take-aways. www.sciencedirect.com/science/arti...

Yesterday, a new paper noted to a flaw w/ systematic reviews an hour later another group launched an AI tool they hope will speed up the process of writing such reviews: w/@christiancao.bsky.social @gengyantang.bsky.social ‪ @webmz.bsky.social @jdwilko.bsky.social www.statnews.com/2025/06/12/r...

Favour request...need a couple of people with trials knowledge to click through the pages of the Promise Delphi survey (not fill it in) and to let me know if it is clear... 15-minute job. I will owe you a (comparable!) favour...ideally today!

There is a difference between an ‘expert’ and ‘a vocal academic with a strong opinion’. Of course, much of the media completely unable to make this distinction.

Starting to look like I might not be able to work at Harvard anymore due to recent funding cuts. If you know of any open statistical consulting positions that support remote work or are NYC-based, please reach out! 😅

‼️HOT OFF THE PRESS‼️ ⭐Updated Responsible AI use in Evidence Synthesis (RAISE) 3-paper collection on OSF: osf.io/fwaud/files/... ⭐ Now includes: 1️⃣Recommendations for practice for the main roles in the ecosystem 2️⃣ Guidance on building & evaluating AI tools 3️⃣Guidance on selecting & using AI tools

Super excited to be presenting on the responsible AI use in evidence synthesis (RAISE) recommendations & guidance this week at #ESMARConf2025 🙌 Come join us! Full details on RAISE here osf.io/fwaud/

When retraction is used as a ‘gold standard’ for the development or validation of trustworthiness tools, my first thought is ‘these people have not been privy to many integrity investigations’. The decision of whether or not to retract a paper is itself a matter of judgement.

Concerns raised about an RCT of Vitamin K2 in JAMA Internal Medicine: www.pubpeer.com/publications...

Following onto this, I feel this requires a philosophical resolution, which is that the mediator not be "pregnancy" but the state of the concept is/fetus. In which case exposure is fully mediated & you can take an IV / Local ATE approach for E, which is the relevant question anyway (?)

Please spread the word! Psych Science is recruiting volunteers to help conduct computational reproducibility checks. If you have experience writing reproducible analysis scripts in psychology, and want to join the team, please apply! www.psychologicalscience.org/publications...

Listening to the Blue Prince soundtrack while marking 100 exam scripts to capture the sense of incremental progress in the face of seemingly endless struggle.

International #ResearchIntegrity conference taking place in beautiful Sydney Australia on 16-18 November 2025 🐨 🌏 Great speakers including @elisabethbik.bsky.social @jdwilko.bsky.social @jasonchin.bsky.social For more info contact @simongandevia.bsky.social 🧪

Describing the "method of randomisation" in your target trial takes some guts

‘What about multicollinearity?’ This is a question asked by peer reviewers with nothing to say. Has anyone written anything about how misguided this question tends to be? Someone should write it if not.

‘The decision to omit non-significant results was a stylistic one’.

experience of in vivo research is that completeness of reporting (ARRIVE) is so limited that risks of bias (SyRCLE) is mostly 'uncertain' - see eg doi.org/10.12688/wel...