SETOL: SemiEmpirical Theory of (Deep) Learning The draft is just about ready Why weightwatcher--and the HTSR theory--work github.com/CalculatedCo...

Where does HTSR and the weightwatcher theory come from ? T𝐡𝐞 𝗪𝐢𝐥𝐬𝐨𝐧 𝐄𝐱𝐚𝐜𝐭 𝐑𝐞𝐧𝐨𝐫𝐦𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐆𝐫𝐨𝐮𝐩 A new principle of learning that is not only fundamental to our understanding of AI 🧠 I have a draft of the theory monograph up on github, and it is just about ready lnkd.in/gBsZ-QKF

🎉 🚀 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡𝐢𝐧𝐠 𝟐𝟎𝟎𝐊 𝐝𝐨𝐰𝐧𝐥𝐨𝐚𝐝𝐬 🥳 💯 𝐖𝐞𝐢𝐠𝐡𝐭𝐖𝐚𝐭𝐜𝐡𝐞𝐫: 𝐃𝐚𝐭𝐚-𝐅𝐫𝐞𝐞 𝐃𝐢𝐚𝐠𝐧𝐨𝐬𝐭𝐢𝐜𝐬 𝐟𝐨𝐫 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 WeightWatcher is based on theoretical research into 𝑾ℎ𝒚 𝑫𝒆𝒆𝒑 𝑳𝒆𝒂𝒓𝒏𝒊𝒏𝒈, using the new 𝐓𝐡𝐞𝐨𝐫𝐲 𝐨𝐟 𝐇𝐞𝐚𝐯𝐲-𝐓𝐚𝐢𝐥𝐞𝐝 𝐒𝐞𝐥𝐟-𝐑𝐞𝐠𝐮𝐥𝐚𝐫𝐢𝐳𝐚𝐭𝐢𝐨𝐧 (HTSR), published in JMLR, Nature Comm., and NeurIPS weightwatcher.ai

Reminder: tomorrow at 10AM PST we will be finishing the table read of section 4.2 of the SETOL monograph Here's the video from last week www.youtube.com/watch?v=0WhB... and the latest version of the paper can be found in ⁠theory-paper

SETOL: SemiEmpirical Theory of (Deep Learning) & the connection to Renormalization Group Turns out, AI models obey a fundamental law of physics when they are trained well. A big thanks to the ML Research Jam for giving me the opportunity to present. www.slideshare.net/slideshow/se...

"We have not used perturbation theory—we have used an axe on the Hamiltonian" Ken Wilson ( Nobel Prize, Physics 1982 )

The weightwatcher theory paper is just about ready. SETOL: SemiEmpirical Theory of (Deep) Learning It's been a passion project of mine for nearly 10 years. before submitting it, I'd like to have a few people read it carefully and comment

How did come up with the idea to look for power law signatures in deep learning models ? Why does power law behavior matter in neural systems and deep learning? Here’s the story:

The quality of a NN layer has an Effective Free Energy obtained through a volume-preserving change of measure analogous to taking a single step of the Wilson Exact Renormalization Group For an ideal layer (alpha=2), this can be experimentally validated with weightwatcher

You can see the emerging signatures of the Wilson Exact Renormalization Group in the best trained layers of modern LLMs like Llama and Falcon. If you're an old physics supernerd like me, that's super cool. But it's also super useful for AI people. weightwatcher.ai

calculatedcontent.com/2024/12/24/w...

A quick weightwatcher workup on the new Falcon3 base models. As predicted by theory, the average weightwatcher layer alpha systematically decrease with increasing model size. The exception is the 10B model, which is an upscaled model.

The theory behind weightwatcher is essentially an application of the Wilson Renormalization Group. The PL exponent alpha=2 is the analogous critical exponent separating the good generalization and overfit (i.e., spin-glass) phases of the NN layer.

Why alpha=2 is the ideal state of a NN layer ? In our upcoming monograph, A SemiEmpirical Theory of (Deep) Learning, we show that the HTSR metrics can be derived as an phenomenological Effective Hamiltonian, but one that is governed by a scale-invariant partition function, just like the Wilson RG

Llama3-70B is Baking

updated history.

What is a SemiEmpirical Theory. Second pass. Comments ? Suggestions ?

What is a SemiEmpirical Theory ?

The SETOL layer quality metric, derived from statistical mechanics, correlated perfectly with the HTSR alpha layer quality metric. Good sign

OLMo-7B. Looks pretty good!

The weightwatcher theory (SETOL) posits that the quality of a NN layer is given by a sum of the integrated R-transform R(z), over the power law tail of the ESD. When alpha=2, the Inverse Wishart model is a good model of the ESD, and the branch cut starts right at the tail.

Breaking down silos: Researchers highlight Nobel-winning AI breakthroughs and call for interdisciplinary innovation techxplore.com/news/2024-11... #AI #Physics #NobelPrize

I recently worked up a bunch of examples of Instruction Fine-Tuned models using weightwatcher. I hope they are useful to you. weightwatcher.ai/models.html

Most statistical mechanics is quite simple. Mean theory plus perturbation (i.e., Replica calculation ) in a real-world system, however, the correlations are nontrivial so you have to treat them phenomenologically

The recent Physics and Chemistry Nobel Prizes, AI, and the convergence of knowledge fields www.cell.com/patterns/ful...

new, large open-weights model out of China which compares to, even beats, Llama3.1 405B. Meet Hunyuan-Large by Tencent: a 389B param MOE (52B active). It is the largest open-source Transformer-based MoE model in the industry paper: arxiv.org/pdf/2411.02265

weightwatcher.ai #LLMs #LargeLanguageModels #AI #DNN #NeuralNetworks #FineTuning #InstructionFineTuning #DeepLearning #DeepNeuralNetworks

In our work on the HTSR Theory and the new SETOL SemiEmpirical Theory of (Deep) Learning, we have discovered that the generalization performance of NNs are governed by a kind of Universality and a related Conservation Principle.

#LLM #LargeLanguageModel #AI

WeightWatcher (w|w) is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN), without needing access to training or even test data. Here are over a dozen new examples of popular Instruction Fine-Tuned models. weightwatcher.ai/models.html