...and here's part 2: what I actually did. Turns out you could use an SFP+ 10GBASE-T module to make a (very small) cup of tea: www.gilesthomas.com/2026/04/10g-...

WiFi has made home wired networking boring for the last 20 years or so, I think, but 10Gb/s "fixes" that :-) I've had a bit of fun over the last two weeks doing an upgrade, here's part 1 of a two-part writeup: www.gilesthomas.com/2026/04/10g-...

The appendices in "Build an LLM (from Scratch)" have a lot of really useful stuff. Should I have read them before heading off on my own training runs? I think no -- it would have saved time, but by trying, failing, and trying again, I learned more. YMMV! www.gilesthomas.com/2026/04/llm-...

Time to wrap up my "Interventions" series: www.gilesthomas.com/2026/04/llm-...

Some interesting results with instruction fine-tuning. In general you'd expect that a model with lower loss on your test set would fine-tune to do better instruction-following. But the correlation doesn't seem as close as I would have thought: www.gilesthomas.com/2026/04/llm-...

I had 57 checkpoints from my last local LLM training run on my disk, and thought it would be interesting to use them to show how the model got more coherent over time. By 1/3 of the way through, it was surprisingly solid! www.gilesthomas.com/2026/04/how-...

Training a new GPT-2-small-style base model on my RTX 3090 in 40 hours: gradient accumulation, along with some of the other interventions I've experimented with, got me almost (but not quite) all the way to the quality of the original OpenAI model! www.gilesthomas.com/2026/04/llm-...

After spending two months trying out different interventions on my GPT-2-style model, it was time to try stacking them up. Interesting results! www.gilesthomas.com/2026/04/llm-...

I wanted to see whether my results when testing interventions to my GPT-2-style training loop were signal or noise. The results were promising! www.gilesthomas.com/2026/04/llm-...

My final intervention test, in which I discover that there is such a thing as a free lunch, and it's called AMP. www.gilesthomas.com/2026/04/llm-...

Weight tying, by contrast with weight decay, was actually really easy! It didn't help, though :-( www.gilesthomas.com/2026/03/llm-...

Weight decay is conceptually simpler than I worried it might be, but still pretty fiddly to get right... www.gilesthomas.com/2026/03/llm-...

Learning rates for LLMs turned out to be a deep topic, but making some tweaks certainly seems to help my base model train: www.gilesthomas.com/2026/03/llm-...

Now this was a surprise. QKV bias is not meant to be useful -- but with my GPT-2 small model, it looks like it is! www.gilesthomas.com/2026/02/llm-...

Does removing dropout improve our baseline model's test loss? Yes, absolutely, and much more than gradient clipping did. www.gilesthomas.com/2026/02/llm-...

First "intervention" test: does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected. www.gilesthomas.com/2026/02/llm-...

Back to my LLM from scratch series. I want to train the *best* GPT-2-style model that I can locally in two days, and there are various levers to pull. Working out which ones work means I need a baseline for comparison. www.gilesthomas.com/2026/02/llm-...

I wanted to get a custom LLM up onto the @hf.co Hub, and couldn't find an in-depth tutorial. Here's the one I wish I'd found before I got started: www.gilesthomas.com/2026/01/cust...

I thought it would be good to share the base models I've been training on Hugging Face, and now they are :-) www.gilesthomas.com/2026/01/llm-...

I wanted to dig into why the results I got on instruction fine-tuning for each of my models didn't seem to match up well with the loss on the test set. Got some interesting results: www.gilesthomas.com/2026/01/2026...

Having trained a GPT-2 scale base model from scratch in 48 hours locally, I wanted to see if I could do the same faster and at a reasonable cost in the cloud. I could! www.gilesthomas.com/2026/01/llm-...

I managed to train my own base model from scratch on an RTX 3090! Very detailed notes here: www.gilesthomas.com/2025/12/llm-...

So, what's left to do in my series on building an LLM from scratch? And what follow-up series should I work on? Some musings: www.gilesthomas.com/2025/11/llm-...

The end of the beginning... running evals on our model using Llama 3 is the last part of the main body of @sebastianraschka.com's "Build an LLM (from scratch)". Here's my writeup: www.gilesthomas.com/2025/11/llm-...

Back on track with chapter 7 of "Build an LLM (from scratch)": notes on instruction fine-tuning of our GPT-2 model: www.gilesthomas.com/2025/10/llm-...

Back when I started messing with LLMs, it looked to me like you could get reasonably OK results for chat applications without instruction fine-tuning. So before getting into Chapter 7 of "Build an LLM (from scratch)", I decided to see if that was really true: www.gilesthomas.com/2025/10/llm-...

And the next step -- a code walkthrough of my PyTorch version of Karpathy's 2015-vintage RNNs. www.gilesthomas.com/2025/10/retr...

Chapter 6 was easy and fun! Fine-tuning an LLM for classification tasks, with some initially disappointing results -- but it all came out in the wash: www.gilesthomas.com/2025/10/llm-...

Part 22 is live: we finally train the LLM :-) Following @sebastianraschka.com's book, we train on Edith Wharton, then swap in GPT-2 (124M) weights for comparison. Notes on seeding, AdamW, temperature and top-k. www.gilesthomas.com/2025/10/llm-...

Decided to learn about some "retro" language models in parallel with LLMs; here's my first post on that -- Revisiting Karpathy’s 'The Unreasonable Effectiveness of Recurrent Neural Networks'. www.gilesthomas.com/2025/10/revi...

Next up in my LLM from scratch series, some serious yak shaving on something @sebastianraschka.com covers in a sidebar: perplexity. www.gilesthomas.com/2025/10/llm-...

Back to the main track of my LLM from scratch posts: cross entropy -- what it is and why we use it. www.gilesthomas.com/2025/10/llm-...

Part 3, and this was a fun one to write: How do LLMs work? From token IDs through to logits -- projections, matrix multiplications, and attention step-by-step. www.gilesthomas.com/2025/09/how-...

Part 2: "The maths you need to start understanding LLMs": www.gilesthomas.com/2025/09/math...

Part 1: "What AI chatbots are actually doing under the hood". www.gilesthomas.com/2025/08/what...

I wanted to do a "what I've learned so far" post to wrap up my notes on Chapter 4 of @sebastianraschka.com's "Build an LLM (from scratch)" but when it got to 6,000 words I suspected it was getting a bit long. So here's a review of upcoming attractions: www.gilesthomas.com/2025/08/llm-...

Now it's time to look into shortcut connections in @sebastianraschka.com's "Build an LLM (from scratch)" -- where the Talmud becomes a surprisingly useful metaphor! www.gilesthomas.com/2025/08/llm-...

Some thinking in public: if the FFNs in an LLM only work on one context vector at a time, why aren't they bitten by the fixed-length bottleneck? www.gilesthomas.com/2025/08/the-...

After a summer break, it's on to the feed-forward layer in @sebastianraschka.com's "Build an LLM (from scratch)" -- in which I discover that attention is not all you need. www.gilesthomas.com/2025/08/llm-...

The next step in my slow-but-steady progress through @sebastianraschka.com's "Build an LLM (from scratch)" -- what is layer normalisation, why do we do it, and how does it work? www.gilesthomas.com/2025/07/llm-...

Connect your AI coding assistant directly to PythonAnywhere! It works with Claude, Copilot, Cursor, and any MCP-compatible tool. See github.com/pythonanywhe...

New blog post, just documenting a bit of cruft-removal: porting old Fabric3 code to modern Fabric: www.gilesthomas.com/2025/06/fabr...

After 14 years of leading @pythonanywhere.com -- 11 as an independent company and 3 as part of @anacondainc.bsky.social, it's time for me to take a much-needed break, and today was my last day. It's been an amazing ride and I'd like to thank everyone that made it possible.

Moving onwards with @sebastianraschka.com's "Build an LLM (from scratch)". The last step in an LLM, going from context vectors to next-word prediction, looks insanely simple. How can it possibly work? www.gilesthomas.com/2025/05/llm-...

First impressions of Claude 4: vibes seem fixed. 3.5 was likeable, 3.7 less so. So far this seems like a return to form.

We've got Streamlit working on PythonAnywhere with our experimental website system! help.pythonanywhere.com/pages/Stream... -- The bad news is that you need a paid account, Streamlit uses up a *lot* of disk space :-(

Some thoughts (and wild speculation) about scaling attention in LLMs, in my new post working through @sebastianraschka.com's "Build a Large Language Model (from Scratch)" www.gilesthomas.com/2025/05/llm-...

In my latest post on @sebastianraschka.com's "Build a Large Language Model (from Scratch)", I realise that attention heads are dumb -- and that's why they work so well. www.gilesthomas.com/2025/05/llm-...

What better use of a public holiday than learning how multi-head attention works? More notes from reading through @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...

Another attack of overthinking when going through the section on batches in @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...