Some pitfalls in vibe-coding seen along the way: - Unchecked, cursor would have deleted all code calculating gradients and hessians, breaking everything. - The first version completely skipped the obvious get/set_extra_state PyTorch methods, creating problems. Possibly these are newer methods?

Speaking at @scipyconf.bsky.social was a blast! Based on questions from the audience, I released an update to GBNet that enables saving a loading models to disk. One step closer to V1.0

Just published in JOSS: 'GBNet: Gradient Boosting packages integrated into PyTorch' https://doi.org/10.21105/joss.08047

🔥 Speaker Spotlight: Michael Horrell Don't miss @horrellmt.bsky.social at #SciPy2025 for "GBNet: Gradient Boosting packages integrated into PyTorch." 🧠⚙️ Discover how GBNet bridges powerful boosting methods with deep learning workflows. 🔗 scipy2025.scipy.org

🚀 #SciPy2025 is just under two weeks away, and I’ll be there! 🎙️ Talk: “GBNet: Gradient Boosting Packages Integrated into PyTorch” 🗓️ Wed July 9, 11:25 AM (Room 315) I'll be speaking about GBNet, an Open Source package I maintain.

Bayesian modeling provides a nice test bed for this Eg: When finding a mean, how much to weight the prior (call it Model 1) vs the sample average (Model 2)? Some bayesian math gives the optimal answer given # of obs. A bit of GBNet & PyTorch empirically derives the same answer almost exactly.

Next application for GBNet is basically a data-aware model averaging or a mixture of experts type of analysis. Situation: you have several models with predictions Q: Is there a data-driven way to combine them? And, for convenience, can I use XGBoost to do find the right averaging coefficients?

GBNet calls XGBoost/LightGBM under the hood. This means you can bring native XGB/LGBM features to PyTorch with little effort. Categorical splitting is one interesting feature to play with using GBNet. To scratch the surface, I fit a basic Word2Vec model using XGBoost for categorical splitting.

Just released GBNet V0.5.0 Beyond some usability improvements, the uncertainty estimation for the Forecasting module got merged in. Now GBNet forecasting is: ✅ Faster ✅ More accurate than Prophet ✅ Provides uncertainty estimates ✅ Supports changepoints

One benefit to speeding up model fitting code 5X is that you can use that saved time for other things. Adding conf intervals for gbnet forecasting module, I can do train/validation holdout for this and still be 3-4X faster. Trying to get 80% test coverage: New method: 76% avg Prophet: 55% avg

So it's continually a nice surprise that stuff like this kinda just works. I asked GBNet for a second prediction output. I slapped on torch.nn.GaussianNLLLoss and out comes a variance estimate that is well calibrated.

Just merged changepoints into the forecasting sub-module of GBNet and released V0.4.0. Default forecast performance improved by 20% and achieved a 5X speedup. Using the random training, random horizon benchmark, now 9 of 9 example datasets have better performance with GBNet compared to Prophet.

Still working on changepoints. Several methods work but don't improve benchmarks. When your model is Trend + XGBoost, XGB can just handle a lot of non-stationarity. Most promising method so far (see plot) asks GBDT to fit and find the changepoints. Another cool application of GBNet (see equation).

Claude gets half credit. It added changepoints for the PyTorch trend option but skipped it for the GBLinear option. Unfortunately it's back to the drawing board. PyTorch changepoints fit too slowly and GBLinear, I now realize, can't actually turn them off. It does work though!

Next, I'll add changepoints to the Forecasting model for GBNet. With this feature, Prophet's primary prediction functionality will be available in GBNet. I wonder how much the LLMs will be able to one-shot it. According to the Prophet paper it's just a broken stick regression with lasso penalty.

Just released v0.3.0 of GBNet. New features: - ReadTheDocs website - GBLinear - GBLinear integration into forecasting model (10X faster, 10% improvement in predictions) - Python class refactoring

Case in point: GBNet Forecast fits XB + GBDT(X). It has a linear component. I replaced PyTorch Linear with GBLinear (removing batchnorm) and... 1. improved accuracy (see table) 2. sped up fitting 10X by using fewer training rounds 3. improved worst-case performance (see plot)

The newest addition to GBNet is GBLinear, a linear layer. GBLinear provides the same functionality as PyTorch Linear, BUT the bias and weights are updated via boosting using gradients and hessians. Why do this?

Cool trick I hadn't seen before today -- You can solve Ridge Regression with some simple concatenations to the X and Y inputs. Turned out great for my use-case because I was using a pure least-squares solver but I wanted Ridge Regression.

I ran across "Harmonic Cross-Entropy Loss" this week: github.com/KindXiaoming... Nice to see a deep learning project that is easy to just jump into and start using.

Due to a user request, I added Ordinal Regression into GBNet. You can now use XGBoost/LightGBM for Ord. Regression. Ord. loss is complex and has viable alternatives. But, on a set of 19 Ordinal Datasets, Ord. Reg. using LightGBM came out on top. Maybe worth keeping in mind if your data is Ordinal.

For the latest release of GBNet, I added the LightGBM backend for the forecasting model. Now you can fit Linear Trend + LightGBM with a couple lines of code. Performance-wise, the LightGBM backend wins overall. Over 500 trials: LightGBM wins 236 (47%) XGBoost wins 153 (31%) Prophet wins 111 (22%)

The right is the embedding that, when multiplied by Beta, gives classification logits. To intuit this, consider the direction defined by the 2D columns of Beta (a 2 x 10 matrix) as the specific classification directions.

I used GBNet to fit different 2D embeddings of MNIST. One is trained to classify via GBNet(X) * Beta. The other is trained by contrastive learning via || GBNet(X) - GBNet(Y) ||. Plots are below. Which is the contrastive embedding? Which is the classification embedding?

POV: me flying from 80 degree Miami to -25 wind chill Chicago earlier this week

Maybe the weirdest training loss you'll see today:

Really I'm finding the biggest positive change between twitter and here is that my feed isn't crammed with videos and tiktoks. Definite plus.

Migrating from twitter (likely posting in both locations). I'll be using this platform for connection to the AI and Machine Learning community. What I'm working on right now: github.com/mthorrell/gb...