The PAISS summer school is back with an incredible line of speakers (and more to come). Spread the word !

Come and visit us in Grenoble!

I wish there were a `British' button to language models: optimization --> optimisation, behavior --> behaviour, initialization --> initialisation etc...

Have you ever encountered similar plots? The training makes no progress, followed by a sharp transition where a new “feature” is learnt. And this occurs with a constant stepsize! Spoiler: An incremental Romain Gary is hidden at the end of this thread... arxiv.org/abs/2304.00488

A (not at all) short presentation of the paper "(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability" which can be found on arxiv: arxiv.org/abs/2302.08982