ArXiv affirming human responsibility without telling you what tools to use — and it being largely uncontroversial — is where you can start to feel what modus vivendi will look like.

Given that we suck at not overprojecting onto non-human objects, but we also think ascribing intention to them can sometimes be useful, should we switch to talking about models’ “so-called reasoning” or “fake thinking traces?”

Given that we suck at not overprojecting onto non-human objects, but we also think ascribing intention to them can sometimes be useful, should we switch to talking about models’ “so-called reasoning” or “fake thinking traces?”

See you today at UIC for the IDEAL Workshop on Language Models and Text Elicitation Mechanisms.

My post doesn‘t mistake why the automation of science enterprise gets the nature of science fundamentally wrong. Saying that people production is a function that's hard to ignore is not inconsistent with saying that people must define what is scientifically valuable. Weird to frame it as opposed.

Yay, another paper by Berna and Erkan to dismantle wishful thinking about how clear-cut replicability is as a signal of good science.

I wonder which of the strategies we use to make research more relatable or concrete for people will also boost LLMs’ ability to judge science. Things like showing examples of the task so reader can judge if problem seems real are disincentivized as it is, would be a shame if AI makes that worse.

We have a small — single-digit — number of frontier models and they’re still very similar to each other. But the differences in epistemic & moral temperament between just two of the things are already like a galvanic battery, shedding illuminating light in this post. Imagine when we have hundreds.

New post on AI, self-preserving compassion, and how none of us are immune to it. open.substack.com/pub/jessicah...

pretty much aligned with how i have come to think about this subject but stated way more clearly than the chaos my mind often produces. i enjoy how jessica genuinely engages with all sides of the issue in good faith.

When does using AI to produce research slip into fraud? And why does so much AI-for-science 'optimism' actually rest on a deeply pessimistic view of scientific agency? 🤔 New post: statmodeling.stat.columbia.edu/2026/04/22/f...

Brief fun survey from Jessica, Andrew & myself: If you are a faculty member, research scientist, postdoc, or senior Ph.D. student in any area of science, please take five minutes and fill it out. We’ll share the results widely along with some reflections. docs.google.com/forms/d/e/1F...

In “Six Memos for the Next Millenium”, Italo Calvino distilled six literary virtues he felt should endure regardless of how the world changed. As we enter an age of automated evaluation & production of science, what are the parallel epstemic virtues we should try to preserve? We want your input! 🧵

idk how or why but it's the first time i came across this paper by Xiao-Li Meng where he offers this banger (if you'd rather hear it from someone that isn't me): "replication paradox: the more we replicate, the surer we fool ourselves" 🫡

Academia in a nutshell.

Excited to give a tutorial this year at ICML 2026!

Oh no.

The real metascience nightmare will be when journals start using AI to require that authors actually followed their preregistration. Personally, I can’t wait. So many people having to eat the silly “our preregistered study” claims slapped all over their papers regardless how much they deviated.

This essay captures what AI for science debates often fail to recognize: That in many academic fields, people creation is the main goal. That the supervision IS the science. That the real threat is "a slow, comfortable drift toward not understanding what you're doing" ergosphere.blog/posts/the-ma...

This worries me most about AI in science. It's what happened with the advent of Stata and SPSS. They enabled incredibly rapid advances in the sciences and also set the stage for the replication crisis. Sadly, so many scientists simply don't understand the stats they are using.

I don't think co-authors need to agree on AI usage. It's still on you to read over what co-authors write and decide if you're ok with it. It's still on you to decide what you're willing to put your name on. The only thing that's changed is that there's new ways to be embarassed, ie, obvious AI-isms

When did poetry get so irrelevant & could AI increase demand for experimentation & jaggedness in creative writing? It's kind of striking how abstraction & experimentation in writing have never been celebrated like in visual art. Could this ever change? jessicahullman.substack.com/p/stop-makin...

Looking forward to giving a talk at the Likert symposium today on validating LLM simulations as behavioral evidence, with a bit as well on why treating LLMs purely as human stand-ins for traditional social science misses the bigger picture

last night in my dream, i was taking a graduate class from feyerabend and it was glorious

A good rule of thumb is that when philosophers refer to something as objective it is something fake that doesn't exist, when they refer to something as subjective it is something we definitely know exists, and that they always mean it to be worse and more problematic to appeal to the latter.

At this point, we have useful AI tools for finding issues with papers. We haven’t demonstrated we can remove humans from the loop or eliminate papers. Cowen leaps from useful tool to rewriting history of economics, abolishing AER & economics becoming a branch of software engineering.

Took a break today from more serious writing to consider what LLMs might do for human writing--maybe even free it from making sense, the way photography freed art from reproducing reality. Could poetry become more "violent," and what would that even look like? open.substack.com/pub/jessicah...

"Against The Pathologization Of Scientific Understanding" Preprint of a forthcoming commentary from Erkan & me. Pathologization of many scientific processes and practices has become increasingly popular w the replication crisis. Such pathologies do not accurately represent the scientific process.

7/ In summary: automated AI procedures for empirical work are leaning even MORE on diff-in-diff -- a concerning development. If anything, my view is we are doing too much DiD nowadays for empirical applied causal work.

Some news to keep from my husband

Now seems like a good time to announce that I also raised a secret $100M last year

Great piece by @soumayakeynes.ft.com on AI in economics in today’s FT (link at end) featuring Refine.ink. Indeed, Refine regularly catches issues that need correcting but are missed by even the most rigorous current review processes. a short 🧵 1/

apropos of the cesar chavez thing it is time to revisit this sootyempiric.blogspot.com/p/no-heroes-...

This morning I amazed by students by scoring 19/20 on an AI writing detection task they thought would result in random performance. I may suck at feelings, but my linguistic hypersensitivity more than makes up for it.

Working hypothesis: If you're doing research and don't occasionally have a small existential crisis, either you've been blessed to work in an exceptional field (do tell which one it is!), or maybe you're being a bit naive.

Call for Submissions: AI for Social Science Methodology (Yale) • Keynote: @nachristakis.bsky.social • Panel with editors of leading journals on publishing AI research • Mentoring roundtables for early-career scholars • Generous travel support Discussion-driven, high-quality research.

But please see great complications to this latter concept raised by @jessicahullman.bsky.social: jessicahullman.substack.com/p/living-the...

Hard to grasp such casual dismissal in light of the huge pedogical value of a conversation partner who can dialogue at length about any resource you give it. If your goal is to build understanding of complex material, LLMs will help. To deny this is bizarre. Makes me wonder if theyve tried modern AI

Yup, plenty of examples of adaptation to new tools for fraud detection. Case study: Problem: 2005-2015, journals start screening submissions with text similarity software Solution: run your plagiarized text through a paraphrasing engine (Grammarly, Quillbot), often leaving "tortured phrases".

Was just thinking about this today after chatting w/ @reeserichardson.bsky.social about detecting scientific fraud at scale. I suspect overall misrepresentation of results get harder as agentic eval improves, but fraud industry may persist by chasing the periphery of what automated evaluation can do

There should be a term for when someone presents ”serious” content (prose, slides, etc) with overtly AI stylistic markers as if no one will notice they took it straight from a chatbot. I’m noticing it a lot in an academic context & it always seems so tacky. Maybe, You have AI on your face?

A few people have asked for the syllabus from my grad seminar on Generative AI for social science -- just posted it here: statmodeling.stat.columbia.edu/2026/03/10/n...

I ran this in progressive mode w/Opus4.6 on a paper where refine.ink had given me very useful feedback. It caught many of the same minor issues! Main difference was in high level feedback-with refine it was much closer to what I'd expect from experts. But not bad for a fraction of refine cost! (~$3)

How your email finds me.

Interesting paper. When science is cheap, multiverse analysis becomes the key abstraction for authors & journals to strategize for. These results suggest its the new unit of evidence authoring & review should accommodate. Related to some of my comments here: substack.com/@jessicahull...

Dear hivemind, Do you have any favorite references on failures of LLM-as-a-judge (and universally enforceable validity checks? besides some kind of cohen's kappa on a small subset)

I find the blog post and underlying post very useful. “How realistic is it to expect multiverse to be used widely, given that most authors first and foremost want to convince readers they have a clear point?”

"The idea that you didn’t need to assume rational agents who were easy to model mathematically but could have various agents w deviations from rationality... that had no impact. Why? Because believing those models required an act of faith" Hard to imagine not needing to understand & exercise taste

Perhaps real scientific progress will begin when we stop writing articles with sweeping generalities in the title! More seriously, many have piled on this post, I think it accurately expresses potential but like many viral takes is overly optimistic about what has sustained science thus far… 1/2

Pragmatic & actionable interpretability are buzzwords arguing for mech interp to study concrete tasks. It's the right instinct, but still underspecified. What counts as a concrete task? What's upper bound on performance? What do users need to know? Decision theory has answers!