With all renewed discussion about "Sparse AutoEncoders (#SAE)" as a way of doing #MechanisticInterpretability of #LLMs, I am resharing a part of my PhD where we proved years ago about how sparsity automatically emerges in autoencoding. arxiv.org/abs/1708.03735

Registrations close for #DRSciML by Noon (Manchester time). Do register soon to ensure you get the Zoom links to attend this exciting event on the foundations of #ScientificML 💥

Recently I gave an online talk @ India's premier institute IISc 's "Bangalore Theory Seminars" where I explained our results on size lowerbounds for neural models of solving PDEs via neural nets. #SciML #AI4SCIENCE I cover work by one of my, 1st year PhD student, Sebastien. youtu.be/CWvnhv1nMRY?...

Today is 70th anniversary of the summer meeting at Dartmouth which officially marked the beginning of AI research 💥 Interestingly "Objective 3" in 1955 was already about having theory of neural nets. 🙂 stanford.io/2WJJJGN

I got selected for the "Early Career Highlights" of ACM IKDD International Conference on Data Science (CODS) 2025. Looking forward to the talk at IISER, Pune in December. www.acm.org/articles/acm...

Why does noisy gradient-descent train neural nets? This fundamental question in ML remains unclear. In our hugely revised draft my student @dkumar9.bsky.social gives the full proof that a form of noisy-GD, Langevin Monte-Carlo (#LMC), can learn arbitrary depth 2 nets. arxiv.org/abs/2503.10428

Registrations are now open for the international workshop on foundations of #AI4Science #SciML that we are hosting with Prof. Jakob Zech. In-person seats are very limited, please do register to join online 💥 drsciml.github.io/drsciml/

Please do get in touch if you have published paper(s) on solving singularly perturbed PDEs using neural nets. #AI4Science #SciML

Some luck to be hosted by a Godel Prize winner, Prof. Sebastien Pokutta, and to present our work in their group 💥 Sebastien heads this "Zuse Institute Berlin (#ZIB) " which is an amazing oasis of applied mathematics bringing together experts from different institutes in Berlin.

Interested in statistics? Prof Subhashis Ghoshal will be delivering the below public lecture tomorrow: Title: Immersion posterior: Meeting Frequentist Goals under Structural Restrictions Time: Aug 5 16:00-17:00 Abstract: www.newton.ac.uk/seminar/45562/ Livestream: www.newton.ac.uk/news/watch-l...

Hello #FAU. Thanks for the quick plan to host me and letting me present our exciting mathematics of ML in infinite-dimensions, #operatorlearning. #sciML Their "Pattern Recognition Laboratory" is completing 50 years! @andreasmaier.bsky.social 💥

University of Manchester has a 1 year post-doc position that I am happy to support in our group if you are currently an #EPSRC funded PhD student - and have the required specialization for work in our group. Typicall we prefer candidates who have published in deep-learning theory or fluid theory.

Do mark your calendars for "DRSciML" (Dr. Scientific ML 😉) on September 9 and 10 🔥 drsciml.github.io/drsciml/ - We are hosting a 2 day international workshop on understanding scientific-ML. - We have leading experts from around the world giving talks. - There might be ticketing. Watch this space!

Major ML journals that have come up in the recent years, - dl.acm.org/journal/topml - jds.acm.org - link.springer.com/journal/44439 - academic.oup.com/rssdat - jmlr.org/tmlr/ - data.mlr.press No reason why these cant replace everything the current conferences are doing and most likely better.

So, the next time you train a deep-learning model, it's probably worthwhile to have a baseline for the only provable adaptive gradient deep-learning algorithm - our delta-GClip 🙂

Our insight is to introduce an intermediate form of gradient clipping that can leverage the PL* inequality of wide nets - something not known for standard clipping. Given our algorithm works for transformers maybe that points to some yet unkown algebraic property of them. #TMLR

Our "delta-GCLip" is the *only* known adaptive gradient algorithm that provably trains deep-nets AND is practically competitive. That's the message of our recently accepted #TMLR paper - and my 4th TMLR journal 🙂 openreview.net/pdf?id=ABT1X... #optimization #deeplearningtheory

An updated version of our slides on necessary conditions for #SciML, - and more specially, "Machine Learning in Function Spaces/Infinite Dimensions". Its all about the 2 key inequalities on slides 27 and 33. Both come via similar proofs. github.com/Anirbit-AI/S...

Now our research group has a logo to succinctly convey what we do - prove theorems about using ML to solve PDEs, leaning towards operator learning. Thanks to #ChatGPT4o for converting my sketches into a digital image 🔥 #AI4Science #SciML

It would be great to be able to see a compiles list of useful PDEs that #PINNs struggle to solve - and how would we measure success there. We know of edge-cases with simple PDEs, where PINNs struggle, but then often those aren't the cutting-edge of use-cases of PDEs.

A revised version of our delta-GClip algorithm - which is probably the *only* deep-learning algorithm that provably trains deep-nets while using step-size scheduling - and competes/supersedes heuristics like Adam and even Adam+Clipping on transformers. arxiv.org/abs/2404.08624

He was the first person to interview me for a PhD position in applied maths and stats when I decided to shift my career focus in that direction. Years later when I became a faculty, he accepted my invite to fly to UK from Leipzig to give a talk and meet my students. Sayan is irreplaceable 😶

Fun @ @aifunmcr.bsky.social 😁

My PhD student @dkumar9.bsky.social has done an amazing job of blending multiple deep results to establish that Langevin Monte-Carlo algorithm provably learns 2-layer nets for any data and for any size. This is very rigorously a "beyond NTK" regime. More to come from Dibyakanti 🙂

My first year PhD student Sébastien André-sloan presents at a #INFORMS conference in Toronto. Its a joint work with Matthew Colbrook at DAMTP, Cambridge. We prove a first-of-its-kind size requirement on neural nets for solving PDEs in the super-resolution setup - the natural setup for #PINNs.

doi.org/10.1093/imai... Our first #IMA journal paper! 😊 We showed Gibbs' measures of neural losses can satisfy the Poincare inequality - and this also holds when the loss is non-Lipschitz. This opens up the first way to get convergence of SGD on such nets *without restrictions on data or size*. 💥

second time speaking @ the holy land of statistics, the Indian Statistical Institute (ISI) 🙂 In 2022 I was @ the Bangalore campus. Same topic : theory of operator learning - the story continues 🔥 Both journal papers were published at #TMLR in 2024.

Here are the slides that my PhD student Dibyakanti Kumar made for his talk @ the #CMStatistics conference talk yesterday on our journal, www.linkedin.com/posts/anirbi... #PINNs are a barely understood way of using nets, for solving PDEs - and we take a careful look at them. #AI4SCIENCE #SciML

This plot is from our 3rd #TMLR journal paper of the year - lnkd.in/e9NWcPcK - on generalization bounds for #DeepOperatorNet methods of PDE solving. #AI4Science Our prediction from theory is this: *use Huber loss to solve PDEs* - and here's a demonstrative comparison on Heat PDE

Likely I will have many posts on this - this paper was a huge labour's of love! "Size-Independent Generalization Bounds for DeepONets" #AI4Science openreview.net/pdf?id=21kO0... its a tour-de-force analysis in Rademacher theory for operator learning. thanks to my amazing student, Dibyakanti !

OpenAI must make specific public commitments to ensure their contractors in Africa pay a fair wage with a premium and adequate mental health care for traumatic work. www.cbsnews.com/news/labeler...

Now I am member of the London Mathematical Society #LMS. It was a surprisingly simple application to make - and it went through! Though I still need to pay the membership fees 😅

If you are at the 2nd UK AI Conference do stop by my student, Dibyakanti's poster 😊 Its one of his upcoming works - about why and when does Langevin Monte-Carlo train nets! At its core its an observation about neural losses satisfying the right isoperimetry inequalities as needed for LMC to work.

New challenge :) Another mathematical ML researcher and me would be visiting a local school to tell ~16-18 year olds about what is ML and what we do. Needing to totally differently look at what is we do to distil this mess for school children 😅

interesting arxiv.org/abs/2411.099...

interesting visualization of a classic youtu.be/yKnKUIuuZOc?...

Hello BlueSky! Here to discuss theory of deep-learning and using AI for the classical sciences :)