We also provide the first convergence rate analysis that I'm aware of for stochastic unconstrained Frank-Wolfe (i.e., without weight decay), which directly covers the muon optimizer (and much more)!

🔥 Want to train large neural networks WITHOUT Adam while using less memory and getting better results? ⚡ Check out SCION: a new optimizer that adapts to the geometry of your problem using norm-constrained linear minimization oracles (LMOs): 🧵👇

It was a fun panel. Quite informative.

Timeo professores machinae discendi et dona ferentes.

An illustrated guide to never learning anything

We'll present "SAMPa: Sharpness-Aware Minimization Parallelized" at #NeurIPS24 on Thursday! This is joint work with Thomas Pethick and Volkan Cevher. 📍 Find us at Poster #5904 from 16:30 in the West Ballroom.

Stable model scaling with width-independent dynamics? Thrilled to present 2 papers at #NeurIPS 🎉 that study width-scaling in Sharpness Aware Minimization (SAM) (Th 16:30, #2104) and in Mamba (Fr 11, #7110). Our scaling rules stabilize training and transfer optimal hyperparams across scales. 🧵 1/10

This is joint work with wonderful collaborators @leenacvankadara.bsky.social , @cevherlions.bsky.social and Jin Xu during our time at Amazon. 🧵 10/10

@iclr-conf.bsky.social: Please incorporate this ACL style of feedback for reviewers: aclrollingreview.org/authors#step...

Reviewers take note: 57% of people rejected their own argument when they thought it was someone else's. So take it easy with the criticism.