Want to know what training data has been memorized by models like GPT-4? We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models, without requiring access to 🙅‍♀️ Model weights 🙅‍♀️ Training data 🙅‍♀️ Token probabilities 🧵 (1/5)

I’ve been referring people (esp social science/psych PhD students) to this blog post for years. The headline and opening paragraph are all you really need.

Had a great time presenting our work on LLM-based item difficulty estimation at #NCME . If you’re in Denver and would like to discuss measurement research or just catchup in the next couple of days, let me know 😊

Fantastic, thoughtful work! 👏👏

I'm excited to share our latest work: "Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms." (🧵 1/9) arxiv.org/abs/2504.08804

Wooden Shoe Tulip Festival 📍Portland, Oregon 🇺🇲

ネコ飛出し注意 (neko tobidashi chūi) Means “watch for cats darting out” and I love this sign.

A tricky thing about modern society is that no one has any idea when they don’t die. Like, the number of lives saved by controlling air pollution in America is probably over 200,000 per year, but the number of people who think their life was saved by controlling air pollution is zero.

Did you know: our researchers have developed a suite of resources for A-Level students and teachers? "History & Philosophy of Science in 20 Objects" draws on an incredible array of items from our own collection ft. prompts, questions, videos and more! sway.cloud.microsoft/cEekCFBF5CGF... #histsci

rough (like uff in buff) cough (like off in scoff) drought (like ow in cow) though (like o in no) thought (like aw in saw) through (like oo in woo) Enough.

Hello to all my friends at SPSP seeing this message in a hallway or lobby as you hope you are staring at your phone with enough noticeable intensity to avoid having to interact with anyone

1/3 Tutorial on exploring ecological momentary assessment data is online at AMPPS, with: - Accessible ways to visualize data for better understanding - Models to get some first insights - Further reading boxes for more advanced topics - Reproducible pipeline you can run over your own data

Some of us have been meeting up at SPSP for the last few years. This year marks our fifth gathering. Email one of us if you want to join! Location TBD. @mdehghani.bsky.social @drsanaz.bsky.social @simine.com @dorsaamir.bsky.social

My husband just inadvertently inspired one of the simplest, most relatable XY problem¹ demos I've seen. He asked if I could buy unscented TP 🧻 next time I grocery shop. Knowing he had been getting a cold, I probed: when does the scent become a problem? [1/3] ¹ en.m.wikipedia.org/wiki/XY_prob...

I’ll share a more detailed thread on this work later, but for now, I’m excited to share this preprint with the Blsky community! In this research, we used a range of methodologies including thematic analysis, closed- and open-vocabulary analyses (e.g., LIWC, topic modeling), and prototype approach...

Lots of useful info in this thread if you are backing up public data (whether at OSF or elsewhere)

Also, 😉

This thread shows that #measurement is essentially a series of compromises. We want comparability across time but also want metrics to reflect changes in priorities. We want psychometric properties but also input from stakeholders. We want translatability but also local validity. Can’t have it all

Do you need educational and psychological item response data to do your research? The IRW has 600 item response datasets (more coming!), distributed in a standardized format and ready for analysis.

Here are some sayings we haven’t heard in a while: -“Hold your horses!” -“She really takes the egg.” -“You’re the bee’s knees!” -“Cash Money Records taking over for the 9-9 into 2000.”

Posting again since I’m trying to get my syllabus together: I’m teaching objective assessment of personality next semester. Any strong recs for things to cover and main take away points?

For all intensive purposes, what idioms or colloquialisms do people seem to constantly get wrong?

It’s a great time to donate to Wikipedia donate.wikimedia.org/wiki/Ways_to...

So it turns out the whole age-period-cohort thing goes way back, with super cool graphs being generated *in the 1870s.* Check out Keiding (2011): onlinelibrary.wiley.com/doi/10.1002/...

I have an ambitious undergrad who wants to learn Python on her own over the break. Suggestions for online tutorials?

🚨New Preprint🚨 *Moral Stereotyping in Large Language Models* We show that LLMs are inaccurate in estimating the moral values of diverse cultures: LLMs stereotype cultures in predictable ways. Led by my star Ph.D. student, Aliah Zewail osf.io/preprints/ps...

This looks *extremely* neat. Kristoffer consistently provides exceptionally useful tools and visualizations 🤩

Preprint update alert! Happy to share the latest version of this work on large language models (LLMs) for psychological measurement. Internally known as “What Would Thurstone Do (with an OpenAI subscription)?”, it results for an ongoing interdisciplinary collaboration arxiv.org/abs/2405.07447

I have created an objective measure of whether people understand measurement. It is a single item, “Is there such a thing as an ‘objective measure’?” If you answer “yes”, then objectively you don’t

Have been trying out Positron for a few days—loving it so far! Feels like the right IDE for juggling R and Python in analytic workflows positron.posit.co

💯"Qualitative competencies ... include developing and applying theories of learning, designing instruments, and identifying the social, cultural, historical, and political contexts of measurement. Quant. competencies include statistical inference, computational fluency, and psychometric modeling."💯

Spearman (1937). Still applicable today

Not an academic, but when it comes to measurement, I’d recommend “Historical and Conceptual Foundations of Measurement in the Human Sciences”

Four ways to think about latent variables (inspired by Bollen, 2002)