You had one job!

"AI slop" seems to be everywhere, but what exactly makes text feel like "slop"? In our new work (w/ @tuhinchakr.bsky.social, Diego Garcia-Olano, @byron.bsky.social ) we provide a systematic attempt at measuring AI "slop" in text! arxiv.org/abs/2509.19163 🧵 (1/7)

I did a QA with Quanta about interpretability and training dynamics! I got to talk about a bunch of research hobby horses and how I got into them.

The new Tahoe UI is so bad. Everything just looks so blurred...

Interesting opportunity for a research visit through the Azrieli foundation - azrielifoundation.org/azrieli-fell... Let me know if you're interested!

Organizing a workshop? Checkout our compiled material for organizing one: www.bigpictureworkshop.com/open-workshop (and hopefully we'll be back for another iteration of the Big Picture next year w/ Allyson Ettinger, @norakassner.bsky.social, @sebruder.bsky.social)

I’m excited to share that I'm joining Bar-Ilan University as an assistant professor!

A strange trend I've noticed at #ACL2025 is that people are hesitant to reach out to papers/"academic products" authors. This is unfortunate for both parties! A simple email can save a lot of time to the sender, but is also one of my favorite kind of email as the receiver!

The EPFL NLP lab is looking to hire a postdoctoral researcher on the topic of designing, training, and evaluating multilingual LLMs: docs.google.com/document/d/1... Come join our dynamic group in beautiful Lausanne!

Had a really great and fun time with @yanai.bsky.social, Niloofar Mireshghallah, and Reza Shokri discussing memorisation at the @l2m2workshop.bsky.social panel. Thanks to the entire organising team and attendees for making this such a fantastic workshop! #ACL2025

I had a lot of fun contemplating about memorization questions at the @l2m2workshop.bsky.social panel yesterday together with Niloofar Mireshghallah and Reza Shokri, moderated by @pietrolesci.bsky.social who did a fantastic job! #ACL2025

I'll present our work w/ @santosh-tyss.bsky.social @yanai.bsky.social @barbaraplank.bsky.social on LLMs memorization of distributions of political leanings in their pretraining data! Catch us at L2M2 workshop @l2m2workshop.bsky.social #ACL2025 tmrw 📆 Aug 1, 14:00–15:30 📑 arxiv.org/pdf/2502.18282

Ai2 is excited to be at #ACL2025 in Vienna, Austria this week. Come say hello, meet the team, and chat about the future of NLP. See you there! 🤝📚

It's crazy that people give more than a single invited talk during the same conference (diff workshops). A single talk (done right) is challenging enough.

What if you could understand and control an LLM by studying its *smaller* sibling? Our new paper introduces the Linear Representation Transferability Hypothesis. We find that the internal representations of different-sized models can be translated into one another using a simple linear(affine) map.

I really like this take. Academia and the open source community should embrace transparency in data, even in the cost of the issues that come with it. These issues should be of course studied and documented, but not used as an indicator or a signal to shut down the whole operation.

Prompting is our most successful tool for exploring LLMs, but the term evokes eye-rolls and grimaces from scientists. Why? Because prompting as scientific inquiry has become conflated with prompt engineering. This is holding us back. 🧵and new paper with @ari-holtzman.bsky.social .

What's up with @arxiv-cs-cl.bsky.social Wasn't the entire premise of this website to allow uploading of papers w/o the official peer review process??

Check out our take on Chain-of-Thought. I really like this paper as a survey on the current literature on what CoT is, but more importantly on what it's not. It also serves as a cautionary tale to the (apparently quite common) misuse of CoT as an interpretable method.

𝐇𝐨𝐰 𝐜𝐚𝐧 𝐰𝐞 𝐩𝐞𝐫𝐟𝐞𝐜𝐭𝐥𝐲 𝐞𝐫𝐚𝐬𝐞 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬 𝐟𝐫𝐨𝐦 𝐋𝐋𝐌𝐬? Our method, Perfect Erasure Functions (PEF), erases concepts perfectly from LLM representations. We analytically derive PEF w/o parameter estimation. PEFs achieve pareto optimal erasure-utility tradeoff backed w/ theoretical guarantees. #AISTATS2025 🧵

I'll be at #NAACL2025: 🖇️To present my paper "Superlatives in Context", showing how the interpretation of superlatives is very context dependent and often implicit, and how LLMs handle such semantic underspecification 🖇️And we will present RewardBench on Friday Reach out if you want to chat!

💡 New ICLR paper! 💡 "On Linear Representations and Pretraining Data Frequency in Language Models": We provide an explanation for when & why linear representations form in large (or small) language models. Led by @jackmerullo.bsky.social, w/ @nlpnoah.bsky.social & @sarah-nlp.bsky.social

I'm on my way to ICLR! Let me know if you want to meet and/or hang out 🥳

I might be able to hire a postdoc for this fall in computational linguistics at UT Austin. Topics in the general LLM + cognitive space (particularly reasoning, chain of thought, LLMs + code) and LLM + linguistic space. If this could be of interest, feel free to get in touch!

How many _final_ will we end up with?

I'm curious if someone actually found those useful... All the "feedback" I got was just rephrasing my reviews.

The R1 paper claims it does not require any (mostly) supervised data to enable these capabilities with RL, but it doesn't report any details on its data or how much it contains instruction data. It also doesn't tackle contamination...

Sponsor ACL 🥳 It is an AI conference, even though some say otherwise.

@ #NeurIPS2024 Come say hi!

Come chat with me at #NeurIPS2024 and learn about how to use Paloma to evaluate perplexity over hundreds of domains! ✨We have stickers too✨

If I don't post about being on the job market, am I really on the job market?

We just updated the OLMo repo at github.com/allenai/OLMo! There are now several training configs that together reproduce the training runs that lead to the final OLMo 2 models. In particular, all the training data is available, tokenized and shuffled exactly as we trained on it!

ICLR ACs - save us!

Has anyone figured out how to create a keyboard shortcut in keynote to add animation to an object?

🚨 New Paper 🚨 Can LLMs perform latent multi-hop reasoning without exploiting shortcuts? We find the answer is yes – they can recall and compose facts not seen together in training or guessing the answer, but success greatly depends on the type of the bridge entity (80% for country, 6% for year)! 1/N

interfolio has the incredibly sophisticated feature of saving my statements, so why can't it save my demographic information? I assure you, my US-centric race won't change from one application to another.

I mean, if you ask about my desired position...

Where are the agents who can fill out these applications for me?

Back in the day, I used to go on Twitter while waiting for my RNN models to finish training. These days, it's going on bsky (ok, ok, X as well) while waiting for ChatGPT to re-write my statement.

Dear EiC, Get some sleep, Yours truly, Yanai

Training LLMs still has a way to go in terms of the number of authors (blog.google/outreach-ini...)

Happening now!

11:30 at Brickell Will Merrill will introduce DAWGS and how they can be used for estimating novelty of LLMs

What's In My Big Data is such a great toolkit. Let's you easily look inside big pretraining data for specific queries 👀 You can try looking up your own name or whatever text you want. Link to the demo with ngram viewer: wimbd.apps.allenai.org

I also have more WIMBD stickers! Ping me if you want some

In Miami for #EMNLP2024 My collaborators and I are presenting a bunch of interesting papers! DM me if you wanna hear more about them, chat about life, job market, or other topics.

Is someone organizing a workshop at #ACL 2025 and wants to switch with a #NAACL 2025 slot?

Chantal Shaib reports on syntactic "templates" that LLM's like to repeat: arxiv.org/abs/2407.00211 (w/@yanai.bsky.social and @jessyjli.bsky.social)