Our #NeurIPS2025 paper shows that even comparable monolingual tokenizers have different compression rates across languages. But by getting rid of whitespace tokenization and using a custom vocab size for each language, we can reduce token premiums. Preprint out now!

I feel like there are several blog posts or papers that put forth a research agenda of "making AI research a scientific field" or "advancing the science of AI" or something like that. I'm trouble finding them, does this ring a bell to anyone / does anyone have links to notable examples?

I want to print it out giant and put it everywhere

The new “teen safety” program from OpenAI repeats the same lies that companies and governments have been saying since the internet began. This won't achieve better online safety for kids, but it will suppress individual liberty and promote censorship. openai.com/index/buildi...

the *cato* institute says less than 10% of politically motivated terrorism is caused by leftists. the *cato* institute. more than two-thirds is from the far-right.

How can an imitative model like an LLM outperform the experts it is trained on? Our new COLM paper outlines three types of transcendence and shows that each one relies on a different aspect of data diversity. arxiv.org/abs/2508.17669

How did you learn to present code? Are there resources that you recommend using to help teach people?

On Thursday we will be joining the #UCLA Latino Policy & Politics Institute for their demo of the newly updated Latino Data Hub (LDH), a public bilingual data platform built to democratize access to critical data about #Latino communities across the country. Register here:

Here are a couple of slides that I presented yesterday at #aitechgov about open-weight model risk management.

Thanks to @stellaathena.bsky.social for chatting with me about Deep Ignorance: the new paper/project from Eleuther AI and the UK AISI. Bottom line: Worried AI could teach people to build bioweapons? Don’t teach it how fortune.com/2025/08/14/w...

“Your driver's license contains a ton of somewhat immutable information about you” like your name, address, DOB, and face, EFF’s Lisa Femia told the @thetennesean.bsky.social. It's not like a credit card number that can be replaced if it's leaked.

Are you afraid of LLMs teaching people how to build bioweapons? Have you tried just... not teaching LLMs about bioweapons? @eleutherai.bsky.social‬ and the UK AISI joined forces to see what would happen, pretraining three 6.9B models for 500B tokens and producing 15 total models to study

Amazing work by @lyndamk.bsky.social and @datarescueproject.org to preserve history in the face of fascism. #SaveOurSigns

"It's unclear if this research matters because real users speak English and Chinese" has got to be up there for worst dismissive takes about how multilingual doesn't matter.

I forgot that Scholar does alerts not just for keywords but also for specific people so I made alerts for all my advisees and now I get an email when they have a paper out maybe folks already do this and I'm late to the game but honestly those alerts feel great, esp when it's a long-gone advisee

It really annoys me how people have made prominent careers out of writing a paper with a scary headline about AI risks that would have produced a non-scary headline with a minor tweak to the set-up and/or probably isn't a meaningfully real phenomenon in the first place.

Congrats Moonshot on making a "modified X" license that's still open source. This is also making me wonder about the list of models to hold the title "most powerful open source LLM in the world." GPT-2 > GPT-Neo > GPT-J > FairSeq Dense > GPT-NeoX-20B > MPT-7B > Falcon-40B > ??? > DeepSeek-R1

At #ICML2025 and a fan of EleutherAI? Come find me for laptop stickers!

Another banger from one of the most important interp researchers in the world.

Stop by our discover server tomorrow, Friday June 27th, to hear about @catherinearnett.bsky.social's work!

This continual level of ignorance (although I am tempted to assume bad faith or its rhetorical equivalent) is not only dangerous, but undermines work towards very real problems it seems to have originated against (silicon valley hype, corporate control, environmental concerns, the rising tide of

We got another mention! We have a lot of these @noaa.gov datasets backed up now. Be sure to check out our tracker: baserow.datarescueproject.org/public/grid/... Scientists scramble to save threatened federal research databases pubs.aip.org/physicstoday...

A bunch of papers suggest that if X and Y are independent tasks, we might expect to see "emergent" behavior on "X and Y" or some task that requires first X and then Y. I'm really surprised I can't find any papers that dig into this; it's usually a side comment. Do you know any?

Several other groups have put out openly licensed dataset recently, why is ours better? Ablation studies show trained on Common Pile v0.1 outperform them, matching the performance of models trained on the original Pile and OSCAR, though still falling short of FineWeb

Our pretrained models, Comma v0.1-1T and -2T perform comparably to leading models trained in the same regime. These plots also include Qwen as a SOTA 8B reference, though it saw 36T tokens

We put a lot of work into our metadata, such as having two rounds of manually validating the ToS of websites in Common Crawl, manually identifying trustworthy YouTube channels, and leveraging work by the BigCode Project and @softwareheritage.org to build the openly licensed subset of StackV2.

The project of open science for machine learning only works if we are able to distribute the training data. Openly licensed data lets us do that, under mild conditions. We make sure to provide document-level metadata for authorship, licensing information, links back to the originals, and more.

The Common Pile comprises text from 30 distinct sources, covering a wide variety of domains including research papers, code, books, educational materials, audio transcripts, governmental text, and more. Some of this text is commonplace in AI, but a lot of it is pretty new.

Can you train a performant language model using only openly licensed text? We are thrilled to announce the Common Pile v0.1, an 8TB dataset of openly licensed and public domain text. We train 7B models for 1T and 2T tokens and match the performance similar models like LLaMA 1 & 2

Is there a way to search Google Scholar or Semantic Scholar and get a list of all papers where at least one author has a specific affiliation?

Reading the ToS of a thousand websites is exhausting. Please send coffee and hugs.

The Trump administration is continuing its illegal attacks on the scientific establishment by withholding funding and unilaterally ending contractual obligations. It's not about science or policy or national security. It's about the wholesale destruction of academia via any lever they can reach.

People keep plugging AI "Co-Scientists," so what happens when you ask them to do an important task like finding errors in papers? We built SPOT, a dataset of STEM manuscripts across 10 fields annotated with real errors to find out. (tl;dr not even close to usable) #NLProc arxiv.org/abs/2505.11855

Wow. We've had a number of new followers recently. Welcome to the wild and crazy world that is the rescuing of federal data for the public good. Check us out: www.datarescueproject.org We also like emojis. ❤️🛟

No, I didn't fail to win any NSF grants last year. I made the strategic decision to seek funding elsewhere to protect the financial solvency of my non-profit.

I feel so bad for the immigrant scientists in the US who won’t see a conference in the next 4 years because we’re a scientific pariah state. Especially the PhD students, but also postdocs who never got to meet the global community during COVID.

Today, at 11am ET, @storytracer.org will be giving a live demo on the @mozilla.ai Discord showcasing two Blueprints for creating open datasets: audio transcription using self-hosted Whisper models and document conversion using Docling. Join the event here: discord.com/invite/4jtc8...

After a short era in which people questioned the value of academia in ML, its value is more obvious than ever. Big labs stopped publishing the minute commercial incentives showed up and are relentlessly focused on a singular vision of scaling. Academia is a meaningful complement, bringing... 1/2

It was great to speak to @chronphilanthropy.bsky.social about AI research and how it's challenging yet necessary for non-profits to work in the LLM space. Many think we should leave the field to companies, but non-profits have different values and goals and those are important. shorturl.at/8IlmF

Proud to be at the AI Action Summit representing @eleutherai.bsky.social and the open source community. The focus on AI for the public good is exciting! DM me or @aviya.bsky.social to talk about centering openness, transparency, and public good in the AI ecosystem.

In case you're curious how much of a hellscape X is. I opened it today to get greeted with a porn ad. The censored section is a 20 second video clip of a woman sucking a dick, with audio. It autoplays.

I have a new favorite academic journal.

Discussions of AI training dynamics and alignment are often underpinned by formal or informal appeals to a probability distribution over models. If you've thought about such an argument, this is a must read. It's not often you get to say prior work is off by "millions of orders of magnitude"!

I enjoy the big ML conferences *and* ACL *and* COLM so I have at least 8 deadlines a year and should never have to crunch. In reality I have 8 deadlines a year and I’m in a state of perpetual crunch.

The CDC's political leadership is editing language that they dislike out of academic papers. The primary purpose and de facto consequence of this will be the massive reduction if not complete elimination of discussion of queer people from CDC publications. insidemedicine.substack.com/p/breaking-n...

Is anyone well-read in the DS-R1 tea leaves and feels confidant they know what the distillation method used was? It's not clear to me if they mean "train on data from another model" or something that I'd consider "actually distilling"? My current guess is synthetic CoT?

The American response to DeepSeek falsifies the claims that America shouldn't be an open source leader because China will just take it and build on it. If that was true, the response would be "lol thanks for giving us free results, idiot."

This is a crime. Trump is breaking the law. The NSF is violating its contractual obligations. And because of those crimes committed by the government, researchers will not get paid and on-going scientific research projects will be ruined. You can't just "pause" access to grants.

Obligatory "actually my lab invented test-time-compute" post. In "Stay on topic with Classifier-Free Guidance," we show that CFG enables a model to expend twice as much compute at inference time and match the performance of a model twice as large. arxiv.org/abs/2306.17806

If you've automated much of your code-writing, what models do you use to write code? What does your workflow look like? I haven't found one I'm too happy with (but also haven't been trying that hard)