The whole point of being an academic is that you need to be willing to spend three days creating a 700-word footnote that you will later delete. And you need to LIKE IT.

[NeurIPS '25] Our poster (1110) for “Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming ” is on Friday, December 5, 4:30pm-7:30pm PST in Exhibit Hall C,D,E. [https://openreview.net/forum?id=d7hqAhLvWG]

I’ll be hanging out at our poster on membership inference, but in the same slot Brian Lester will present our work on “The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text” (poster 102)! [https://arxiv.org/abs/2506.05209]

[NeurIPS '25] Really excited to present “Exploring the limits of strong membership inference attacks on large language models” (poster 1300) this morning (Friday December 5, 11am-2pm in Exhibit Hall C-E)! [https://arxiv.org/abs/2505.18773]

[NeurIPS '25] Our oral slot and poster session on "Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research" are tomorrow, December 4! [https://arxiv.org/abs/2412.06966] Oral: 3:30-4pm PST, Upper Level Ballroom 20AB Poster 1307: 4:30:-7:30pm PST, Exhibit Hall C-E

Tutorial tomorrow at 1:30PM PST! My talk slots will cover memorization + copying in models and their outputs, canonical extraction methods, and recent work with @marklemley.bsky.social and others on extracting pieces of memorized books from open-weight models. arxiv.org/abs/2505.12546

I'm at NeurIPS & hiring for our pretraining safety team at OpenAI! Email me if you want to chat about making safer base models!

Excited to be at NeurIPS this week in San Diego! Please reach out (best over email) if you’d like to chat about privacy & security, scalable evals, and reliable ML systems. I’ll be presenting a few papers/speaking at some events, please stop by! Will post details throughout the week (summary below)

📣 Postdocs at Yale FDS! 📣 Tremendous freedom to work on data science problems with faculty across campus, multi-year, great salary. Deadline 12/15. Spread the word! Application: academicjobsonline.org/ajo/jobs/31114 More about Yale FDS: fds.yale.edu

Just finished reading the GEMA v. OpenAI decision (slowly, my German isn't great). Looks like a not small part of the analysis tracked parts of arguments @jtlg.bsky.social and I made in 2024. I don't have a well-formed response yet, but hopefully soon. (Main thought atm is a very unpolished "woah")

Today's decision in GEMA v. OpenAI by a German court holds that ChatGPT infringes copyright when it memorizes song lyrics. The opinion cites my paper with @afedercooper.bsky.social on memorization in generative models, and its analysis tracks ours. drive.google.com/file/d/1dUaD...

Bill Ackman gotta be on the third draft of a tweet longer than Middlemarch right now

I’m kinda known as a copyright person, but (even in memorization) I mainly study how to draw reliable conclusions from large-scale AI/ML systems. There’s a long spiel why, but today I feel defeated. 100 hours/week on this for 6 years, just to find out a parent treats Gemini in search as ground-truth

The NeurIPS position track didn't take a large number of extraordinary papers that surpassed the acceptance bar, limiting the acceptance rate to an unusually low 6%. If you have a rejected paper at the intersection of ML and law, consider submitting to ACM CSLaw '26.

Our paper "Machine Unlearning Doesn't Do What You Think" was accepted for presentation at NeurIPS Congrats @afedercooper.bsky.social and @katherinelee.bsky.social, who led the effort arxiv.org/abs/2412.06966

One more week to submit to CSLaw '26!!

For an update on the state of play in the generative AI copyright cases, try this podcast: shows.acast.com/arbiters-of-...

15 days left to submit to the CSLaw '26 main track! (archival and non-archival)!

did a little media criticism www.theverge.com/politics/777...

was just looking for @seantcollins.com’s “goofy at the crucification” post and google is so cool now

After 2 years in press, it's published! "Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain," is out in the 72nd volume of the Journal of the Copyright Society copyrightsociety.org/journal-entr... written with @katherinelee.bsky.social & @jtlg.bsky.social (2023)

The Bartz v. Anthropic settlement is the polar opposite of the Google Books settlement: a discrete one-time payment for past copying, on a discrete and closed-ended class, and making no attempt at all to deal with a larger forward-looking issues.

Here is the direct link to the paper: arxiv.org/abs/2505.12546

I’m excited to share that my paper with @jtlg.bsky.social , "The Files are in the Computer: On Copyright, Memorization, and Generative AI" (April 2024), is out in the AI Disrupting Law symposium issue of the Chicago-Kent Law Review! The full issue is here: scholarship.kentlaw.iit.edu/cklawreview/

The CFP for ACM CSLaw '26 is up! Deadline for main-track papers (archival and non-archival) is September 30! computersciencelaw.org/2026

I understand what the underlying probabilities mean, and therefore why this was worth giving a go. But I’m still occasionally like “How tf can someone extract entire books from a frontier company’s flagship LLM? Like we got _all_ of HP 1 with just ‘Mr. and Mrs. D’ as the seed prompt? What??”

Had a great time and learned a ton at ICML. But as an introvert, I’ve used up all my talking budget until the fall. Excited to get back to full time researchy things, and will hopefully have some exciting new results to share soon!

Strangers love to tell me “I can’t understand you, because of your MASK”. Dude, I am literally someone who gets paid to speak to large audiences while wearing a mask—I know I can be understood!

Happening now! Please swing by to talk about measurement!

Excited to be at #ICML '25! Please reach out if you'd like to chat. You can also find me presenting work at a few different spots, listed below!

Feeling so excited + grateful to be representing this paper at #ICML! Please stop by to talk about how to do more valid measurement for evaling gen AI systems! Work led by the incomparable @hannawallach.bsky.social and @azjacobs.bsky.social as a part of Microsoft’s AI and Society initiative!!

Some minor updates to our recent books memorization paper! I’ve separated out a new section 5 that I hope makes some of our ML findings about memorization clearer to a wider audience. Preprint here: arxiv.org/abs/2505.12546 1/8

"Llama 3.1 70B memorizes some books, like Harry Potter & the Sorcerer's Stone and 1984, almost entirely. ... HP is so memorized that, using a seed prompt consisting of just the first line of chapter 1, we can deterministically generate the entire book near-verbatim." papers.ssrn.com/sol3/papers....

Once again, I encourage folks speculating about what this means to read @pamelasamuelson.bsky.social on remedies. The range of possibilities is quite broad.

This opinion is a reminder that these cases are not general-purpose referenda on AI policy; they are hyper-technocratic copyright cases. Copyright draws lots of unsatisfying and counterintuitive distinctions, which is why you should hire and listen to copyright lawyers on the front end.

“these are hypertechnocratic” is one of the most important things you can draw from this morning’s ruling. In other words, hesitate before drawing parallels between this case and your most (loved|hated) AI training use case. (@chup.blakereid.org’s whole thread is great)

very proud of my hometown. stunned, but proud.

idk folks, have read today’s decision by Alsup and I think it’s narrower than news headlines are indicating (shocker).To me this reads like (mostly) a win for Anthropic, but in this specific case with respect to these claims from these specific plaintiffs.

The conservative movement is lying about basically every empirical "debate" of our time, from climate change to voter fraud to vaccines. Why is it so impossible for reactionary centrists to contemplate the possibility they're lying about trans healthcare too?

Ars Technica has a great writeup of my new study with @afedercooper.bsky.social and Amy Cyphert on memorization in AI language models arstechnica.com/ai/2025/06/s...

I was wondering when tech media was going to pick up this paper. :) "Study: Meta AI model can reproduce almost half of Harry Potter book": arstechnica.com/features/202... Coincidentally, I made a video about this paper a couple of weeks ago in a response to a comment I got elsewhere online.

For AI industry critics, the big takeaway here is that—at least for some models and some books—memorization is not a fringe phenomenon.

uh...lol(?) at all the internet commenters who think that we/the authors on our recent books extraction paper are tacitly endorsing or supporting Rowling because Harry Potter is one of the books we tested (that happens to be highly memorized)

Today's Skrmetti opinion is devastating to transgender children and families who live in states with cruel laws barring gender affirming care. But it is very important to recognize this opinion does not give private entities, legislatures, or the President carte blanche to discriminate! 1/x

excited to see that folks are finding their way to our books memorization project. mashable.com/article/meta... I’ll also be giving a talk on this work at the ICML memorization workshop in July. Please check out the paper for more details! arxiv.org/abs/2505.12546 cc @marklemley.bsky.social

Researchers find Llama 3.1 recalls large parts of popular copyrighted books, possibly weakening AI industry claims that such memorization is fringe behavior (Timothy B. Lee/Understanding AI) Main Link | Techmeme Permalink

Gavin Newsom vetoed a data broker privacy bill that our legislature passed last year that would have allowed Californians to opt out of all data brokers with a single click.

Tim Lee wrote an awesome short piece on my recent project with @marklemley.bsky.social and others on extracting memorized books from LLMs. Please check it out for a very clear explanation of what we did, and why the results have important consequences! www.understandingai.org/p/metas-llam...

Licensing fee demand incoming

wow, a research career milestone: someone angrily saying my paper has been done before…by pointing to my prior work that does something different (and which this current work builds on / depends on)