We discovered that language models leave a natural "signature" on their API outputs that's extremely hard to fake. Here's how it works 🔍 📄 arxiv.org/abs/2510.14086 1/

I didn't believe when I first saw, but: We trained a prompt stealing model that gets >3x SoTA accuracy. The secret is representing LLM outputs *correctly* 🚲 Demo/blog: mattf1n.github.io/pils 📄: arxiv.org/abs/2506.17090 🤖: huggingface.co/dill-lab/pi... 🧑‍💻: github.com/dill-lab/PILS

I wish the ML community would stop trying to turn every technique into a brand name. Just give the thing a descriptive name and call it what it is. Forced backronyms like this are counter productive.

It appears that the only fonts with optical sizes that work with pdflatex are the computer/latin modern fonts. I would kill for a free pdflatex-compatible Times clone with optical sizes so my small text can look good in ArXiv/conference submissions.

If you are writing a paper for #colm2025 and LaTeX keeps increasing your line height to accommodate things like superscripts, consider using $\smash{2^d}$, but beware of character overlaps.

This project was made feasible by the excellent open-source LLM training library @fairseq2.bsky.social; I highly recommend giving it a look! It made both SFT and DPO a piece of cake 🍰

🧵 Adapting your LLM for new tasks is dangerous! A bad training set degrades models by encouraging hallucinations and other misbehavior. Our paper remedies this for RAG training by replacing gold responses with self-generated demonstrations. Check it out here: https://arxiv.org/abs/2502.10

Putting together an unofficial usc Beamer template, I noticed that the USC style guide lists 4 formats for “cardinal red” but each of them is different: PMS 201 C is #9D2235 CMYK: 7, 100, 65, 32 is #A1003D RGB: 135, 27, 30 is #991B1E HEX: #990000 Is this normal? The CMYK is especially egregious.

In Vancouver for NeurIPS but don't have Taylor Swift tickets? You can still spend the day going through our tutorial reading list: cmu-l3.github.io/neurips2024-... Tuesday December 10, 1:30-4:00pm @ West Exhibition Hall C, NeurIPS

Curious about all this inference-time scaling hype? Attend our NeurIPS tutorial: Beyond Decoding: Meta-Generation Algorithms for LLMs (Tue. 1:30)! We have a top-notch panelist lineup. Our website: cmu-l3.github.io/neurips2024-...

What's that? A fully open LM competitive with Gemma and Qwen*? Happy to have helped a bit with this release (Tulu 3 recipe used here)! OLMo-2 13B actually beats Tulu 3 8B on these evals, making it a SOTA fully open LM!!! (*on the benchmarks we looked at, see tweet for more)

These folks have had a huge impact on my research

#socalnlp is the biggest it's ever been in 2024! We have 3 poster sessions up from 2! How many years until it's a two-day event?? 🤯

This is niche but the LLM360 logo always reminds me of the 2014 iOS game Oquonie

Hottest new research challenge: find the lost LLM head!

Everyone follow Sean! He's been working nonstop to perfect our upcoming NeurIPS tutorial

As "X is all you need" and "Transformers are Y" paper titles have died, I propose that we similarly retire: - X of thought - Chain of Y - "[topic] a comprehensive survey" where [topic] is exclusively post-2022 papers - Claims of reasoning/world model/planning based on private defns of one

Nature wrote a nice article about our work! www.nature.com/articles/d41...

I made a map! Thank you to my 2019 self for providing the code github.com/mattf1n/Reli...

Today I learned you can add a citation link to your GitHub repo. citation-file-format.github.io

Oh my, USC is an empire!

And we're having a great time at #EMNLP2024, come talk to us!

I’m proud of this tikz drawing I made today for our upcoming NeurIPS tutorial on decoding (our paper: arxiv.org/abs/2406.16838)

I'll be presenting "Distributional Properties of Subword Regularization" with @zouharvi.bsky.social and Naoaki Okazaki at #EMNLP. arxiv.org/abs/2408.11443 The idea is that stochastic variants of BPE/MaxMatch produce very biased tokenization distributions, which is probably bad for modeling. #NLP

USC NLP folks are on Bluesky! Follow my amazing colleagues here go.bsky.app/KUwSZ6W

A starter pack for #NLP #NLProc researchers! 🎉 go.bsky.app/SngwGeS

How many times did the 14 LA Metro board members + CEO use their Metro TAP cards in the 4.5 years from 1/1/20 to 6/30/24? 2,259 times. 33.5 taps per person per year. 1112 workdays in period. If CEO tapped 2x/workday, 14 board members in total tapped 35 times in 4.5 years.

Just landed in Philly for COLM where I’ll be presenting my work on extracting secrets from LLM APIs at the Wednesday afternoon poster sesh. Please reach out if you wanna hang and talk about sneaky LLM API hacks, accountability, and the geometry of LLM representations! arxiv.org/abs/2403.09539

Ooh we got the answer from Steven Bird! Sorry to link to Twitter, but the story is worth reading. x.com/StevenBird/s... "I got the idea at SIGMOD'99... I had been living in West Africa and experienced inequities of access... I wanted ours to be a society that made its content free to all."

“Media companies aren't on their workers' side in just the same way that Amazon wasn't on publishers' side. The promise of a new, broader copyright [for AI] is like the promise of DRM: it's a prison pretending to be a fortress” pluralistic.net/2024/03/13/h...

Please help me, probabilists of the internet: what is the probability that an unfair m-sided dice rolls its most-likely value more times than any of its other values after n rolls?

We are pleased to announce that the first Conference on Language Modeling will be held at the University of Pennsylvania in Philadelphia at the Zellerbach Theatre. Thanks so much to UPenn CS as well as Mark Yatskar and Zachary Ives for facilitating the amazing venue.

Cross posting to tell you that I am looking forward to speaking (virtually) at Stanford NLP seminar this thursday! I’ll be talking about Trustworthy NLP. When: 01/25 Thurs 11am PT Non-Stanford affiliates registration form (closed at 9am PT on the talk day): forms.gle/j8DQDDvgyLTSZB…

FINALLY 🤖https://aclweb.org/adminwiki/images/5/56/ACL_Anonymity_Policy.pdf

I decided what we need to make blueskAI happen is a feed. Reply here to get added to the whitelist! Whitelisted users can post to the feed by adding the following keywords to a post: 🤖 bskAI blueskAI

Utah is hiring tenure-track/tenured faculty & a priority area is NLP!  Please reach out over email if you have questions about the school and Salt Lake City, happy to share my experience so far.  utah.peopleadmin.com/postings/154...

A cute TikZ diagram for your enjoyment: (I spent too much time making this for a presentation)

Nucleus and top-k sampling are ubiquitous, but why do they work so well? We explain the theory and give a method to address model errors at their source (the softmax bottleneck). 📄 arxiv.org/abs/2310.01693 🧑‍💻 github.com/mattf1n/basi... 1/🧵

Highly doubt any followers on here are planning to apply for GRFP, but I want to support the X alternative 😂 so... If you know anyone planning to apply for NSF GRFP this cycle, please pass on my statements, CV, and feedback from my successful app for a DL/ML/Speech ECE GRFP app from 2019 to them!