I wrote a new thing over at Macroscience for @andrewgerard.bsky.social about predicting scientific breakthroughs. If you missed it yesterday, go check it out and let me know what you think: open.substack.com/pub/macrosci...

Fascinating talk on the history of peer review from Melinda Baldwin at @icssi.org, with some parallels to current crises. Also I hadn't realized peer review was such an American thing.

Sharing our latest endeavour here. How reproducible is your paper? @philipjakobbln.bsky.social and I built rigor.me to ease the burden of computational reproducibility. If you provide a paper, data and code, we execute it and tell you what works (and what fails). Beta is available now: rigor.me

Lol, a paper that describes my life. My natural reaction to feeling like I'm finally getting the hang of something has always been to drop it and try something else. And somehow I still often feel like I'm not getting out fast enough.

"The more phil of science I read, and the more familiar I became with different pockets of the metascience community, the more I came to realize how little consensus there is about how to evaluate science. But I don’t blame those who default to thinking the metascientists have figured things out.

Check out our #FAccT2026 tutorial tomorrow on Bridging Predictions and Interventions in Social Systems! We're going to be building a predictions - interventions index to track ADS systems, evaluations, and gaps therein. Bring a device; I'll try to bring worksheets too :)

Amazing https://load-bear.ing

Using AI to support peer review seems unavoidable, but what quality checks should AI implement? We can take some lessons from metascience on the hard reward design problem that is AI review. I wrote a paper synthesizing a few points the emerging lit seems at risk of confusing. 1/

Excellent essay by Leif, Tyler, and Ben. Interestingly, reification of “internal states” is one of the pitfalls of the intentional stance of Dennett, something we are seeing a lot of AI interoperability research succumbing to.

After two days with Claude Fable 5 the best way I can describe it is "relentlessly proactive" - here's an example where I dropped in a screenshot of a bug and it span up custom CORS Python servers and used pyobjc-framework-Quartz to capture screenshots simonwillison.net/2026/Jun/11/...

It's interesting how quick we are to assume science is aligned by default. There's an idea that good science doesn't require human involvement, it's about the "Truth" which can be discovered & verified independent of our judgment. In doing so we ignore most of the history of science.

Lots of people with strong opinions about use of AI detectors to filter what papers we review (or what stories we consider for awards). I wrote up a toy model of using AI detection to infer author type to get at some implict assumptions we make when we argue that AI detection is or is not useful

the year 2026 in a nutshell

Thoughts on metascientific consequences of AI-generated slides & ideas diluting the impression that speakers are commited to what they present. Science runs on personal attachment more than we admit. If it were a cake mix, how wouldn we add back an egg? statmodeling.stat.columbia.edu/2026/05/28/w...

The influx of first-time, single author, AI-assisted work suggests that these new entrants to the field would benefit from some mentoring about what constitutes a research contribution in AI/ML. How should the community help them get this mentoring? end/

At @arxiv.bsky.social, we are receiving a new type of paper that I call an "I did this experiment" paper. These papers typically report some experiment with an LLM or LLM "agentic" workflow. They are the kind of experiments an "insider" engineer would run to optimize a system. 1/

Some thoughts on the blog today on "humans are unreliable narrators too" as a common defense of AI mental-state language, and what I think it misses about the nature of reasoning or thinking or belief or intention as we understand these terms statmodeling.stat.columbia.edu/2026/05/23/t...

ArXiv affirming human responsibility without telling you what tools to use — and it being largely uncontroversial — is where you can start to feel what modus vivendi will look like.

Given that we suck at not overprojecting onto non-human objects, but we also think ascribing intention to them can sometimes be useful, should we switch to talking about models’ “so-called reasoning” or “fake thinking traces?”

Given that we suck at not overprojecting onto non-human objects, but we also think ascribing intention to them can sometimes be useful, should we switch to talking about models’ “so-called reasoning” or “fake thinking traces?”

See you today at UIC for the IDEAL Workshop on Language Models and Text Elicitation Mechanisms.

My post doesn‘t mistake why the automation of science enterprise gets the nature of science fundamentally wrong. Saying that people production is a function that's hard to ignore is not inconsistent with saying that people must define what is scientifically valuable. Weird to frame it as opposed.

Yay, another paper by Berna and Erkan to dismantle wishful thinking about how clear-cut replicability is as a signal of good science.

I wonder which of the strategies we use to make research more relatable or concrete for people will also boost LLMs’ ability to judge science. Things like showing examples of the task so reader can judge if problem seems real are disincentivized as it is, would be a shame if AI makes that worse.

We have a small — single-digit — number of frontier models and they’re still very similar to each other. But the differences in epistemic & moral temperament between just two of the things are already like a galvanic battery, shedding illuminating light in this post. Imagine when we have hundreds.

New post on AI, self-preserving compassion, and how none of us are immune to it. open.substack.com/pub/jessicah...

pretty much aligned with how i have come to think about this subject but stated way more clearly than the chaos my mind often produces. i enjoy how jessica genuinely engages with all sides of the issue in good faith.

When does using AI to produce research slip into fraud? And why does so much AI-for-science 'optimism' actually rest on a deeply pessimistic view of scientific agency? 🤔 New post: statmodeling.stat.columbia.edu/2026/04/22/f...

Brief fun survey from Jessica, Andrew & myself: If you are a faculty member, research scientist, postdoc, or senior Ph.D. student in any area of science, please take five minutes and fill it out. We’ll share the results widely along with some reflections. docs.google.com/forms/d/e/1F...

In “Six Memos for the Next Millenium”, Italo Calvino distilled six literary virtues he felt should endure regardless of how the world changed. As we enter an age of automated evaluation & production of science, what are the parallel epstemic virtues we should try to preserve? We want your input! 🧵

idk how or why but it's the first time i came across this paper by Xiao-Li Meng where he offers this banger (if you'd rather hear it from someone that isn't me): "replication paradox: the more we replicate, the surer we fool ourselves" 🫡

Academia in a nutshell.

Excited to give a tutorial this year at ICML 2026!

Oh no.

The real metascience nightmare will be when journals start using AI to require that authors actually followed their preregistration. Personally, I can’t wait. So many people having to eat the silly “our preregistered study” claims slapped all over their papers regardless how much they deviated.

This essay captures what AI for science debates often fail to recognize: That in many academic fields, people creation is the main goal. That the supervision IS the science. That the real threat is "a slow, comfortable drift toward not understanding what you're doing" ergosphere.blog/posts/the-ma...

This worries me most about AI in science. It's what happened with the advent of Stata and SPSS. They enabled incredibly rapid advances in the sciences and also set the stage for the replication crisis. Sadly, so many scientists simply don't understand the stats they are using.

I don't think co-authors need to agree on AI usage. It's still on you to read over what co-authors write and decide if you're ok with it. It's still on you to decide what you're willing to put your name on. The only thing that's changed is that there's new ways to be embarassed, ie, obvious AI-isms

When did poetry get so irrelevant & could AI increase demand for experimentation & jaggedness in creative writing? It's kind of striking how abstraction & experimentation in writing have never been celebrated like in visual art. Could this ever change? jessicahullman.substack.com/p/stop-makin...

Looking forward to giving a talk at the Likert symposium today on validating LLM simulations as behavioral evidence, with a bit as well on why treating LLMs purely as human stand-ins for traditional social science misses the bigger picture

last night in my dream, i was taking a graduate class from feyerabend and it was glorious

A good rule of thumb is that when philosophers refer to something as objective it is something fake that doesn't exist, when they refer to something as subjective it is something we definitely know exists, and that they always mean it to be worse and more problematic to appeal to the latter.

At this point, we have useful AI tools for finding issues with papers. We haven’t demonstrated we can remove humans from the loop or eliminate papers. Cowen leaps from useful tool to rewriting history of economics, abolishing AER & economics becoming a branch of software engineering.

Took a break today from more serious writing to consider what LLMs might do for human writing--maybe even free it from making sense, the way photography freed art from reproducing reality. Could poetry become more "violent," and what would that even look like? open.substack.com/pub/jessicah...

"Against The Pathologization Of Scientific Understanding" Preprint of a forthcoming commentary from Erkan & me. Pathologization of many scientific processes and practices has become increasingly popular w the replication crisis. Such pathologies do not accurately represent the scientific process.

7/ In summary: automated AI procedures for empirical work are leaning even MORE on diff-in-diff -- a concerning development. If anything, my view is we are doing too much DiD nowadays for empirical applied causal work.

Some news to keep from my husband

Now seems like a good time to announce that I also raised a secret $100M last year

Great piece by @soumayakeynes.ft.com on AI in economics in today’s FT (link at end) featuring Refine.ink. Indeed, Refine regularly catches issues that need correcting but are missed by even the most rigorous current review processes. a short 🧵 1/

apropos of the cesar chavez thing it is time to revisit this sootyempiric.blogspot.com/p/no-heroes-...