Update on Error Report #2: The impact of acoustic stimulation during sleep on memory and sleep architecture: A meta-analysis The article has been retracted by agreement with the journal editor and authors #metascience

"Meta-meta-analyses trade precision & rigor for expediency. If your aim is rigor, there’s no substitute for doing the hard work." Preach! I once read a #meta-analysis that mistook SE for SD. Since then I am suspicious of all such studies. Do the hard work! matthewbjane.github.io/blog-posts/b...

This is very nice from @matthewbjane.bsky.social as to why umbrella reviews are basically useless (sorry!), essentially because they compound up the issues with the meta analyses they include. I have the same concerns but articulate them less eloquently. matthewbjane.github.io/blog-posts/b...

I've moaned about meta-analyses of meta-analyses for a while and had several interesting experiences reviewing them... Now I can save time and just send people to Matt's post to explain why not to do them.

Happy Friday everyone! I just posted what I think is an important blog post on my website. It is a critique of meta-meta-analyses: meta-analyses of meta-analyses. Link: matthewbjane.github.io/blog-posts/b... #stats #metascience

Happy Friday everyone! I just posted what I think is an important blog post on my website. It is a critique of meta-meta-analyses: meta-analyses of meta-analyses. Link: matthewbjane.github.io/blog-posts/b... #stats #metascience

I'm calling for a total and complete shutdown of JAMA Pediatrics until we can figure out what is going on goodscience.substack.com/p/the-resear... I mean, come ON

(#metascience) A good article by @stuartbuck.bsky.social on the Fluoride and IQ meta-analysis: goodscience.substack.com/p/the-resear.... Summarizes some of the points in our forensic review of it (see the forensic review as a pre-print here: osf.io/preprints/os...). Very nice excerpt below:

When a STALT error messes up your pagination.

Error Report #4: matthewbjane.quarto.pub/meta-analysi... The meta-analysis on Fluoride and IQ by Taylor et al (2025) is methodologically flawed and has data integrity issues. The full report by me, @jamesheathers.bsky.social, and @drg.bsky.social is on OSF: osf.io/preprints/os... #metascience

Error Report #4: matthewbjane.quarto.pub/meta-analysi... The meta-analysis on Fluoride and IQ by Taylor et al (2025) is methodologically flawed and has data integrity issues. The full report by me, @jamesheathers.bsky.social, and @drg.bsky.social is on OSF: osf.io/preprints/os... #metascience

Spent the past couple days looking at meta-analyses in body comp literature to get reasonable parameter estimates to use in simulations for a study we're planning and really trying to avoid giving myself a to-do list of meta-analyses that I need to correct/redo 🙃 cc: @matthewbjane.bsky.social

livefreeordichotomize.com by @lucystats.bsky.social matthewbjane.github.io/blog.html by @matthewbjane.bsky.social www.andrewheiss.com/blog/ by the king @andrew.heiss.phd ngreifer.github.io/blog/ by yours truly (can't resist a self-promotion 😉) Stats blogs are the best!

Coding RCTs for a meta-analysis…

The meta-analysis on the relationship between Fluoride and IQ by Taylor et al. (2025) that RFK Jr. has cited contains major methodological flaws and data integrity issues that we describe in the following pre-print (submitted to meta-psychology): osf.io/preprints/os...

Another great meta-science contribution by @matthewbjane.bsky.social and others. This and his re-analysis of the “social media causes harm” meta-analysis are great examples of how we can use science to serve society without necessarily having to be “activists”

The notorious MBJ and his crew are at it again! 🔥 🔥 #RStats #PyData #Stats #EconSky #DataBS

Most studies in the meta-analysis are no more detailed than the following: “Area A has low fluoride water levels, Area B has high fluoride water levels, and area A has a higher mean IQ” No matching or balancing of covariates. Distinct geographic areas differ more than just their fluoride levels.

They chose to use the standardized mean difference in IQ between a reference and exposure group however the reference in many studies has more fluoride exposure than the exposure group in other studies! There is no consistent definition for the groupings.

One major finding in our pre-print is that the journal ‘Fluoride’ publishes much larger effects than any other academic journal. Fluoride is published by the International Society for Fluoride Research which is run by anti-fluoride activists and hosts water defluoridation workshops. #metascience

I saw @epiellie.bsky.social comment on this so I am tagging her here #metascience

The meta-analysis on the relationship between Fluoride and IQ by Taylor et al. (2025) that RFK Jr. has cited contains major methodological flaws and data integrity issues that we describe in the following pre-print (submitted to meta-psychology): osf.io/preprints/os...

In probability the following is true by definition Pr(sample space) = 1 Pr(empty set) = 0

👀

We just published a paper on the performance of various tests and confidence intervals for the scale coefficients in location-scale models in meta-analysis: doi.org/10.3758/s134... Overall, REML-based LRTs and profile-likelihood CIs performed best, but the number of studies typically must be large.

Error Report #2 (#metascience): t.co/9xyqIVUele The meta-analysis by Stanyer et al. (2022) contains severe data extraction errors. Due to these errors, the meta-analytic results for memory retention scores are incorrect. Reported effects for WPAL outcomes are 5x larger than the corrected effects.

Error report #1 (#metascience): An influential meta-analysis on mindfulness based stress reduction with ~2,900 citations appears to have severe data extraction and methodological errors that inflate the difference in effects between MBSR & control groups: matthewbjane.quarto.pub/meta-analysi...

indeed who gave these people taking time to post helpful comments on published research for no reward the power to be largely ignored by the authors of the commented research and the journals the research was published in?!?!

Someone talk me down from the idea to call X a “conditioning variable” #stats

When you say something is an assumption (e.g., linear regression assumes conditional normality) you need to specify what properties are actually being preserved with this assumption. 1/3

1: #AcademicSky #PsychSciSky #LatinX/einSTEM #QueerSTEM #Science Please share! I'm running a survey on how *U.S.* academics (grad students, faculty, admin., etc) think and feel about recent executive orders affecting academia (e.g., diversity, funding). Here iu.co1.qualtrics.com/jfe/form/SV_...

Here's my entry: Hydroxycholoquine treats COVID-19 because a p-value is small.

I’ve heard stories about profs whose BIGGEST FINDING was made by some student clicking through their data in SPSS. I’m coming to believe that being bad at stats must have been adaptive in that field during a certain historical time.

Only 3 of the 14 original results reported as statistically significant were successfully replicated, with an average replication effect size of 2.9% of the original estimates. TWO POINT NINE PERCENT papers.ssrn.com/sol3/papers....

Pre-prints are a run-of-the-mill modern open science practice. I’d rather a more transparent review process than less. A completely open review process holds reviewers and authors accountable. Let’s not pretend that a blind review is “impartial” and “objective”. #metaresearch

Chopped all my hair off

Just saw skimmed a simulation paper and I knew what the results would be by just looking at the authors.

A silly little thing in Excel that might be useful for forensic data analysts. Have a look at these numbers. Notice anything? /1

Since clearly.app is up. Thank you all for including me in so many starter packs! Really appreciate it

Not sure which of the 60+ data, stats, and ml starter packs to start with? I combined them all and ranked people by how many packs and how many distinct topics they appeared in. These are the 149 people on Bluesky with the broadest importance across quant communities in November, 2024.

🥰

Matt’s are so back

Academic fraud is rife with #china at the forefront. As I write in the Spectator, journals are waking up to reality of fake science thanks to dogged detective work of @elisabethbik.bsky.social worthy winner of @einsteinberlin.bsky.social award #fraud #science www.spectator.co.uk/article/the-...

A list of all the stats/modeling/ML/data starter packs I've seen (26+ and counting):

I’m a bluesky OG

I had no idea meta analysis guru @matthewbjane.bsky.social is on 🦋 woo hoo!!!

#stats #metascience Change scores don’t make any sense here. Anyone have any ideas what the discrepancy could be?

I was wondering why it seemed as if many people's first action when they arrive here is to resume following me from Twitter. It turns out it's not my magnetic personality, it's that they're using Sky Follower Bridge to rebuild their follows. You probably should, too!

A largely non exhaustive pack for meta research and open science. Please circulate it ;) go.bsky.app/R8RNJgc