...

Does labeling AI content affect its persuasiveness? Our latest policy brief evaluates the impact of authorship labels on how people perceive AI-written policy messages. @jurafsky.bsky.social @robbwiller.bsky.social @federicobianchi.bsky.social hai.stanford.edu/policy/label...

It is great to see Gov Spencer Cox stepping up to turn the temperature down and focus ppl on grieving the tragic killing of Charlie Kirk. Gov Cox has long promoted civility across party lines. Our research finds that his messages are persuasive when Americans see it. 👇

We live in a world that feels more divided than ever. This week, we revisit an episode with sociologist Robb Willer and discuss how to bridge even the greatest divides. hiddenbrain.org/podcast/us-2...

Saturday's No Kings protests were massive, inclusive, and nonviolent. It was an honor to team up with @owasow.bsky.social to analyze how nonviolent resistance can wins hearts, minds, and elections.

Check out our new WP on the capacity for LLMs to perform deep canvassing! Fantastic leadership by @tomcostello.bsky.social. Doesn't hurt to have @gordpennycook.bsky.social and @dgrand.bsky.social on the team, either.

Take a 👀 at this policy brief from @stanfordhai.bsky.social on how AI agents can test ideas in social science. Honored to be part of this amazing team: @joon-s-pk.bsky.social @cqzou.bsky.social @aaronshaw.bsky.social @mako.cc Carrie Cai, Meredith Ringel Morris, Percy Liang, @mbernst.bsky.social

I had a great time this week talking about: ⚡️effective strategies for social change ⚡️political persuasion ⚡️managing coalitional divides ⚡️utopian versus pragmatic political worldviews with the great Lauren Harper Pope of the Welcome PAC!

With the new school year, my lab (@pascl-stanford.bsky.social) is adjusting to some big departures, as many folks move on to exciting new homes. Here’s a look at this stellar crop of alums & where they’re headed now. It’s been a privilege to work with such talented scholars & good friends!

🚨 ONLY 5 DAYS LEFT! 🚨 Submit your ideas for structural democratic reforms by 9/15. It should take less than 5 minutes to share an idea via our online form. Submit as many ideas as you wish and submit anonymously, if you prefer!

🚨Submissions for the Structural Democratic Reforms Project are still open! Please submit your ideas for reforms to strengthen and safeguard our democracy by 9/15!🚨 (Note: it's easy to submit an idea....could take just a few minutes. Pls spread the word!) More info...🧵👇

@mdehghani.bsky.social @maxhuibai.bsky.social @mohammadatari.bsky.social @lmesseri.bsky.social @sabdurah.bsky.social

H/t also to some of the many scholars whose work we drew on: @kurtjgray.bsky.social @chrisbail.bsky.social @lpargyle.bsky.social @johnjhorton.bsky.social @mbernst.bsky.social @schoeneggerphil.bsky.social @mjcrockett.bsky.social

*Major* kudos to Luke Hewitt and @ashwinia.bsky.social (who co-led the research), and to @isaiasghezae.bsky.social. And thanks to @pascl-stanford.bsky.social and @stanfordpacs.bsky.social for generously supporting this project.

To explore further, you can use LLM-simulated participants to generate predicted experimental effects using this demo! 👇 treatmenteffect.app

This capacity has several applications for science and practice – e.g., running low-cost pilots to identify promising interventions, or simulating experiments that may be harmful to participants – but also limitations and risks, including concerns about bias, overuse, & misuse.

Overall our results show high accuracy of LLM-derived predictions for experiments with human participants, generally greater accuracy than samples of lay and expert humans. Paper👉: docsend.com/view/qeeccug...

Finally, we find LLMs can accurately predict effects on socially harmful outcomes, such as the impact of antivax FB posts on vax intentions (Jennifer Allen et al., 2024). This capacity may have positive uses, such as for content moderation, though also highlights risks of misuse.

We also evaluated predictive accuracy for “megastudies,” studies comparing the impact of a large number of interventions. Across nine survey and field megastudies, LLM-derived predictions were modestly accurate (Notably, accuracy matched or surpassed expert forecasters)

Despite known training data inequalities, LLM-derived predictive accuracy was comparable across subgroups. However, there was little heterogeneity in experimental effects we studied, so more research is needed to assess if/how LLM predictions of experimental results are biased.

Important work finds biases in LLM responses resulting from training data inequalities. Do these biases impact accurate prediction of experimental results? To assess, we compare predictive accuracy for: ➡️women & men ➡️Black & white participants ➡️Democrats & Republicans

But what if LLMs are simply retrieving & reproducing known experimental results from training data? We find evidence against this: analyzing only studies *unpublished* at time of GPT4’s training data cut-off, we find high predictive accuracy (r = .90, adj. r = .94).

We also find that predictive accuracy improved across generations of LLMs, with GPT4 surpassing predictions elicited from an online sample (N = 2,659) of Americans. Paper 👉 treatmenteffect.app/paper.pdf

We prompted the model with (a) demographic profiles drawn from a representative dataset of Americans, and (b) experimental stimuli. The effects estimated by pooling these responses were strongly correlated with the actual experimental effects (r = .85; adj. r = 0.91)!

To evaluate predictive accuracy of LLMs for social science experiments, we used #GPT4 to predict 476 effects from 70 well-powered experiments, including: ➡️50 survey experiments conducted through NSF-funded TESS program ➡️20 additional replication studies (Coppock et al. 2018)

🚨New WP: Can LLMs predict results of social science experiments?🚨 Prior work uses LLMs to simulate survey responses, but can they predict results of social science experiments? Across 70 studies, we find striking alignment (r = .85) between simulated and observed effects 🧵👇

🚨🚨Call for Submissions🚨🚨 For the “Structural Democratic Reforms Project” we are gathering ideas from a diverse range of academics, practitioners, and concerned citizens. Submit your ideas for reforms to improve our political system NOW! democratic-reforms.org/submit 1/

🚨🚨Do Americans accurately perceive union benefits? 🚨🚨Our new paper (led by Jonne Kamphorst) out this week at PNAS finds: Americans significantly and systematically underestimate material benefits associated with union benefits. 🧵👇 www.pnas.org/doi/10.1073/...

🚨WP🚨 We test 9 online samples and find clear tradeoffs between attentiveness and representativeness - which sample is best depends on research q and priorities. For social/political qs I rec Bovitz/Lucid, for complex designs I rec Cloud/Prolific osf.io/preprints/ps...

I had the great pleasure of chatting w Shankar Vedantam on Hidden Brain discussing political persuasion, empathy, moral reframing, and my unforgivable behavior towards roommates during my early 20's. Thanks Shankar and the Hidden Brain team for the fun conversation! hiddenbrain.org/podcast/us-2...

🚨Join us for a workshop on how to run megastudies at #SPSP2024 🚨 Jan Voelkel and I will share our experiences and recommendations for conducting crowdsourced megastudies. Please sign up 👇: spsp.wufoo.com/forms/spsp-2...

Sharing this super-cool initiative from the UC Berkeley Center for Human-compatible AI: The Prosocial Ranking Challenge, which awards $60k to five winning algorithms on real users and real platforms using a browser extension. humancompatible.ai/news/2024/01...

Please consider applying for this new post-doc position!

📢Come join the Polarization and Social Change Lab at Stanford University📢 Our lab has an open Research Coordinator position! We invite applicants from psychology, poli sci, sociology, econ, data science, et al. Position begins in Spring or Summer of 2024. Please share to all who may be interested!

🚨🚨We are thrilled to announce the winners of the inaugural @pascl-stanford.bsky.social Bridging Divides & Strengthening Democracy Field Test Grants!🚨🚨