Feb 3, 2025 - We started fighting to save our data. July 3, 2025 - We launched #SaveOurSigns with Minn librarians. April 2026 - We are still talking about the importance of public data as a public good. ❤️🛟

Regretfully, the story about LLMs anti-polarizing people was not real.

If this is real, it’s very plausibly the biggest win for alignment research.

You have a moral imperative to refuse to work with these people or develop models for these purposes.

How do you identify which problems are interesting and valuable? When people don’t work on problems that matter, why do you think that is?

If I was going to claim that a finetuning methodology for machine unlearning “really worked,” what evidence would you like to see?

Language identification still proves to be a challenging task, especially for web data. In collaboration with @mlcommons.org @eleutherai.bsky.social @jhu.edu and 97 community members, we created CommonLID, a new benchmark for LangID for 100+ languages!

Has anyone else had Claude code become non-functional recently? Even with a test input it spins for minutes without doing anything. Same thing happens in terminal.

And the next talk (exact details TBA) by @pjox.bsky.social and @very-laurie.bsky.social from Common Crawl on work we've been collaborating on to build better benchmarking of LangID systems and understand the issues with the long tail of human language that comes up at Common Crawl scales.

We’re bringing back a Community Spotlight talk series, highlighting cool work being done by members of our community. We’re kicking it off with a talk on running diffusion-based world-models in real time on consumer hardware. Jan 9th at 2 pm US Eastern Time

What are people's favorite paper / project websites? I'm looking to build a library to base future ones I make off of. The one EleutherAI has done that I'm proudest of is deepignorance.ai

If you're looking for the 60 minutes piece on CECOT that Bari Weiss canned to protect the President, you can find it here: archive.org/details/60-m...

Incredibly proud of my friend and colleague @storytracer.com. Two weeks ago he and his cofounders @sucho-org.bsky.social were honored for organizing a global network of volunteers to exfiltrate and back up endangered Ukrainian cultural heritage in the wake of the invasion by Russia.

Really great to see NVIDIA staking out a pro-open data position. This used to be common, if not the norm in AI, and the backing away from this level of transparency has done a lot of harm to the research community.

Compare the statement about the antisemitic terror attack in Australia by the Israeli Prime Minister with the one by Zohran Mamdani, and ask yourself who more truly cares about condemning antisemitism, as opposed to using it to promote unrelated politics.

"The incentives made me do it" is an excuse, not a justification. You can be better than that, and if you're not it's because you choose to not be.

It's a wrap on EvalEval in San Diego! A jam packed day of learning, making new friends, critically examining the field of evals, and walking away with renewed energy and new collaborations! We have a lot of announcements coming, but first: EvalEval will be back for #ACL2026!

In 2023-ish it was trendy to write papers trying to explain why scaling laws had power law structures. The papers I remember were pretty unconvincing. Did anything meaningful come of this work? What does the best work in this vein look like?

Put this person in jail. Put the person who drafted it in jail. Put the higher ups who covered it up in jail. This is a crime. If Kilmar had done the same they wouldn't hesitate to punish him. ICE is a criminal organization, not a law enforcement organization, and justice requires accountability.

Hyped to write "The models in this paper cost us 476,246.57 USD to train. I'm sorry you are sad we didn't redo all of our experiments on multiple independent training datasets. If you'd like to give us a million dollars we'd be happy to run the experiments you wish" in my response to a reviewer.

🚨 AI keeps scaling, but social impact evaluations aren’t–and the data proves it 🚨 Our new paper, 📎“Who Evaluates AI’s Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations,” analyzes hundreds of evaluation reports and reveals major blind spots ‼️🧵 (1/7)

Our #NeurIPS2025 paper shows that even comparable monolingual tokenizers have different compression rates across languages. But by getting rid of whitespace tokenization and using a custom vocab size for each language, we can reduce token premiums. Preprint out now!

I feel like there are several blog posts or papers that put forth a research agenda of "making AI research a scientific field" or "advancing the science of AI" or something like that. I'm trouble finding them, does this ring a bell to anyone / does anyone have links to notable examples?

I want to print it out giant and put it everywhere

The new “teen safety” program from OpenAI repeats the same lies that companies and governments have been saying since the internet began. This won't achieve better online safety for kids, but it will suppress individual liberty and promote censorship. openai.com/index/buildi...

the *cato* institute says less than 10% of politically motivated terrorism is caused by leftists. the *cato* institute. more than two-thirds is from the far-right.

How can an imitative model like an LLM outperform the experts it is trained on? Our new COLM paper outlines three types of transcendence and shows that each one relies on a different aspect of data diversity. arxiv.org/abs/2508.17669

How did you learn to present code? Are there resources that you recommend using to help teach people?

On Thursday we will be joining the #UCLA Latino Policy & Politics Institute for their demo of the newly updated Latino Data Hub (LDH), a public bilingual data platform built to democratize access to critical data about #Latino communities across the country. Register here:

Here are a couple of slides that I presented yesterday at #aitechgov about open-weight model risk management.

Thanks to @stellaathena.bsky.social for chatting with me about Deep Ignorance: the new paper/project from Eleuther AI and the UK AISI. Bottom line: Worried AI could teach people to build bioweapons? Don’t teach it how fortune.com/2025/08/14/w...

“Your driver's license contains a ton of somewhat immutable information about you” like your name, address, DOB, and face, EFF’s Lisa Femia told the @thetennesean.bsky.social. It's not like a credit card number that can be replaced if it's leaked.

Are you afraid of LLMs teaching people how to build bioweapons? Have you tried just... not teaching LLMs about bioweapons? @eleutherai.bsky.social‬ and the UK AISI joined forces to see what would happen, pretraining three 6.9B models for 500B tokens and producing 15 total models to study

Amazing work by @lyndamk.bsky.social and @datarescueproject.org to preserve history in the face of fascism. #SaveOurSigns

"It's unclear if this research matters because real users speak English and Chinese" has got to be up there for worst dismissive takes about how multilingual doesn't matter.

I forgot that Scholar does alerts not just for keywords but also for specific people so I made alerts for all my advisees and now I get an email when they have a paper out maybe folks already do this and I'm late to the game but honestly those alerts feel great, esp when it's a long-gone advisee

It really annoys me how people have made prominent careers out of writing a paper with a scary headline about AI risks that would have produced a non-scary headline with a minor tweak to the set-up and/or probably isn't a meaningfully real phenomenon in the first place.

Congrats Moonshot on making a "modified X" license that's still open source. This is also making me wonder about the list of models to hold the title "most powerful open source LLM in the world." GPT-2 > GPT-Neo > GPT-J > FairSeq Dense > GPT-NeoX-20B > MPT-7B > Falcon-40B > ??? > DeepSeek-R1

At #ICML2025 and a fan of EleutherAI? Come find me for laptop stickers!

Another banger from one of the most important interp researchers in the world.

Stop by our discover server tomorrow, Friday June 27th, to hear about @catherinearnett.bsky.social's work!

This continual level of ignorance (although I am tempted to assume bad faith or its rhetorical equivalent) is not only dangerous, but undermines work towards very real problems it seems to have originated against (silicon valley hype, corporate control, environmental concerns, the rising tide of

We got another mention! We have a lot of these @noaa.gov datasets backed up now. Be sure to check out our tracker: baserow.datarescueproject.org/public/grid/... Scientists scramble to save threatened federal research databases pubs.aip.org/physicstoday...

A bunch of papers suggest that if X and Y are independent tasks, we might expect to see "emergent" behavior on "X and Y" or some task that requires first X and then Y. I'm really surprised I can't find any papers that dig into this; it's usually a side comment. Do you know any?

Several other groups have put out openly licensed dataset recently, why is ours better? Ablation studies show trained on Common Pile v0.1 outperform them, matching the performance of models trained on the original Pile and OSCAR, though still falling short of FineWeb

Our pretrained models, Comma v0.1-1T and -2T perform comparably to leading models trained in the same regime. These plots also include Qwen as a SOTA 8B reference, though it saw 36T tokens

We put a lot of work into our metadata, such as having two rounds of manually validating the ToS of websites in Common Crawl, manually identifying trustworthy YouTube channels, and leveraging work by the BigCode Project and @softwareheritage.org to build the openly licensed subset of StackV2.

The project of open science for machine learning only works if we are able to distribute the training data. Openly licensed data lets us do that, under mild conditions. We make sure to provide document-level metadata for authorship, licensing information, links back to the originals, and more.

The Common Pile comprises text from 30 distinct sources, covering a wide variety of domains including research papers, code, books, educational materials, audio transcripts, governmental text, and more. Some of this text is commonplace in AI, but a lot of it is pretty new.

Can you train a performant language model using only openly licensed text? We are thrilled to announce the Common Pile v0.1, an 8TB dataset of openly licensed and public domain text. We train 7B models for 1T and 2T tokens and match the performance similar models like LLaMA 1 & 2