We'll present our work, "CHAMELEON: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning," at #ICML2025! This is joint work with Francesco Tonin and @CevherLIONS. 📍 Find us at Poster E-2807 from 11 AM today. Excited to connect and discuss!

arxiv.org/abs/2502.07529 🚀 Key results: - Based on conditional gradient method - Beats Muon+Adam on NanoGPT (tested up to 3B params) - Zero-shot learning rate transfer across model size - Uses WAY less memory (just one set of params + half-precision grads) - Provides explicit norm control

🔥 Want to train large neural networks WITHOUT Adam while using less memory and getting better results? ⚡ Check out SCION: a new optimizer that adapts to the geometry of your problem using norm-constrained linear minimization oracles (LMOs): 🧵👇

We'll present "SAMPa: Sharpness-Aware Minimization Parallelized" at #NeurIPS24 on Thursday! This is joint work with Thomas Pethick and Volkan Cevher. 📍 Find us at Poster #5904 from 16:30 in the West Ballroom.