Came to #NeurIPS2024 for the research news, but staying for these incredible views. I am presenting some recent works that (I think) significantly advance the discourse on LLM memorization, training data detection; & a study on hallucinations x model collapse in diffusion models.

if you're a PhD student at CMU doing AI/ML, lmk if you want to be added to this starter pack. (I don't belong in this list, but I don't know how to remove myself from this pack 😂) go.bsky.app/9APVxQQ

🚀New Paper: Active Data Curation Effectively Distills Multimodal Models arxiv.org/abs/2411.18674 Smol models are all the rage these days & knowledge distillation (KD) is key for model compression! We show how data curation can effectively distill to yield SoTA FLOP-efficient {C/Sig}LIPs!! 🧵👇

How to drive your research forward? “I tested the idea we discussed last time. Here are some results. It does not work. (… awkward silence)” Such conversations happen so many times when meetings with students. How do we move forward? You need …

1/Open LLM evals often face data contamination concerns. Private curators (like ScaleAI) have addressed this with private + expert evaluations. We argue that this shift poses new risks including financial incentives & eval bias. w/ @hbxnov.bsky.social 📝: pratyushmaini.github.io/blog/2024/ri... 🧵

Medically adapted foundation models (think Med-*) turn out to be more hot air than hot stuff. Correcting for fatal flaws in evaluation, the current crop are no better on balance than generic foundation models, even on the very tasks for which benefits are claimed. arxiv.org/abs/2411.04118

Temporally shifted data splits in membership inference can be misleading ⚠️ Be cautious when interpreting these benchmarks!

1/6 A lot of us are grappling with peer review these days, but its worst manifestation is when prestigious conference awards overlook critical flaws. Case in point: #EMNLP2024 ’s Best Paper Award. I & @iamgroot42.bsky.social wrote a blog on what went wrong: www.anshumansuri.com/blog/2024/ca... 🧵

1/5 Earlier this year, I joined @datologyai.com to give wings to the data research I had been doing in academia. Today, I am absolutely thrilled to share what we’ve been working on! Techvember Ep 2: How we made the #1 LLM Pre-training Data Recipe. Blog: 👉 tinyurl.com/best-llm-data 🧵

my new found guilty pleasure is watching the new reasoning models struggle by think-maxxing them with questions from JEE Advanced

pretty excited about tomorrow's class. we will know the winner of our first red-blue team pokemon unlearning challenge. 620 more battles to go ⚔️