I wanted to see whether my results when testing interventions to my GPT-2-style training loop were signal or noise. The results were promising! www.gilesthomas.com/2026/04/llm-...

My final intervention test, in which I discover that there is such a thing as a free lunch, and it's called AMP. www.gilesthomas.com/2026/04/llm-...

Weight tying, by contrast with weight decay, was actually really easy! It didn't help, though :-( www.gilesthomas.com/2026/03/llm-...

Weight decay is conceptually simpler than I worried it might be, but still pretty fiddly to get right... www.gilesthomas.com/2026/03/llm-...

Learning rates for LLMs turned out to be a deep topic, but making some tweaks certainly seems to help my base model train: www.gilesthomas.com/2026/03/llm-...

Now this was a surprise. QKV bias is not meant to be useful -- but with my GPT-2 small model, it looks like it is! www.gilesthomas.com/2026/02/llm-...

Does removing dropout improve our baseline model's test loss? Yes, absolutely, and much more than gradient clipping did. www.gilesthomas.com/2026/02/llm-...

First "intervention" test: does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected. www.gilesthomas.com/2026/02/llm-...

Back to my LLM from scratch series. I want to train the *best* GPT-2-style model that I can locally in two days, and there are various levers to pull. Working out which ones work means I need a baseline for comparison. www.gilesthomas.com/2026/02/llm-...

I wanted to get a custom LLM up onto the @hf.co Hub, and couldn't find an in-depth tutorial. Here's the one I wish I'd found before I got started: www.gilesthomas.com/2026/01/cust...

I thought it would be good to share the base models I've been training on Hugging Face, and now they are :-) www.gilesthomas.com/2026/01/llm-...

I wanted to dig into why the results I got on instruction fine-tuning for each of my models didn't seem to match up well with the loss on the test set. Got some interesting results: www.gilesthomas.com/2026/01/2026...

Having trained a GPT-2 scale base model from scratch in 48 hours locally, I wanted to see if I could do the same faster and at a reasonable cost in the cloud. I could! www.gilesthomas.com/2026/01/llm-...

I managed to train my own base model from scratch on an RTX 3090! Very detailed notes here: www.gilesthomas.com/2025/12/llm-...

So, what's left to do in my series on building an LLM from scratch? And what follow-up series should I work on? Some musings: www.gilesthomas.com/2025/11/llm-...

The end of the beginning... running evals on our model using Llama 3 is the last part of the main body of @sebastianraschka.com's "Build an LLM (from scratch)". Here's my writeup: www.gilesthomas.com/2025/11/llm-...

Back on track with chapter 7 of "Build an LLM (from scratch)": notes on instruction fine-tuning of our GPT-2 model: www.gilesthomas.com/2025/10/llm-...

Back when I started messing with LLMs, it looked to me like you could get reasonably OK results for chat applications without instruction fine-tuning. So before getting into Chapter 7 of "Build an LLM (from scratch)", I decided to see if that was really true: www.gilesthomas.com/2025/10/llm-...

And the next step -- a code walkthrough of my PyTorch version of Karpathy's 2015-vintage RNNs. www.gilesthomas.com/2025/10/retr...

Chapter 6 was easy and fun! Fine-tuning an LLM for classification tasks, with some initially disappointing results -- but it all came out in the wash: www.gilesthomas.com/2025/10/llm-...

Part 22 is live: we finally train the LLM :-) Following @sebastianraschka.com's book, we train on Edith Wharton, then swap in GPT-2 (124M) weights for comparison. Notes on seeding, AdamW, temperature and top-k. www.gilesthomas.com/2025/10/llm-...

Decided to learn about some "retro" language models in parallel with LLMs; here's my first post on that -- Revisiting Karpathy’s 'The Unreasonable Effectiveness of Recurrent Neural Networks'. www.gilesthomas.com/2025/10/revi...

Next up in my LLM from scratch series, some serious yak shaving on something @sebastianraschka.com covers in a sidebar: perplexity. www.gilesthomas.com/2025/10/llm-...

Back to the main track of my LLM from scratch posts: cross entropy -- what it is and why we use it. www.gilesthomas.com/2025/10/llm-...

Part 3, and this was a fun one to write: How do LLMs work? From token IDs through to logits -- projections, matrix multiplications, and attention step-by-step. www.gilesthomas.com/2025/09/how-...

Part 2: "The maths you need to start understanding LLMs": www.gilesthomas.com/2025/09/math...

Part 1: "What AI chatbots are actually doing under the hood". www.gilesthomas.com/2025/08/what...

I wanted to do a "what I've learned so far" post to wrap up my notes on Chapter 4 of @sebastianraschka.com's "Build an LLM (from scratch)" but when it got to 6,000 words I suspected it was getting a bit long. So here's a review of upcoming attractions: www.gilesthomas.com/2025/08/llm-...

Now it's time to look into shortcut connections in @sebastianraschka.com's "Build an LLM (from scratch)" -- where the Talmud becomes a surprisingly useful metaphor! www.gilesthomas.com/2025/08/llm-...

Some thinking in public: if the FFNs in an LLM only work on one context vector at a time, why aren't they bitten by the fixed-length bottleneck? www.gilesthomas.com/2025/08/the-...

After a summer break, it's on to the feed-forward layer in @sebastianraschka.com's "Build an LLM (from scratch)" -- in which I discover that attention is not all you need. www.gilesthomas.com/2025/08/llm-...

The next step in my slow-but-steady progress through @sebastianraschka.com's "Build an LLM (from scratch)" -- what is layer normalisation, why do we do it, and how does it work? www.gilesthomas.com/2025/07/llm-...

Connect your AI coding assistant directly to PythonAnywhere! It works with Claude, Copilot, Cursor, and any MCP-compatible tool. See github.com/pythonanywhe...

New blog post, just documenting a bit of cruft-removal: porting old Fabric3 code to modern Fabric: www.gilesthomas.com/2025/06/fabr...

After 14 years of leading @pythonanywhere.com -- 11 as an independent company and 3 as part of @anacondainc.bsky.social, it's time for me to take a much-needed break, and today was my last day. It's been an amazing ride and I'd like to thank everyone that made it possible.

Moving onwards with @sebastianraschka.com's "Build an LLM (from scratch)". The last step in an LLM, going from context vectors to next-word prediction, looks insanely simple. How can it possibly work? www.gilesthomas.com/2025/05/llm-...

First impressions of Claude 4: vibes seem fixed. 3.5 was likeable, 3.7 less so. So far this seems like a return to form.

We've got Streamlit working on PythonAnywhere with our experimental website system! help.pythonanywhere.com/pages/Stream... -- The bad news is that you need a paid account, Streamlit uses up a *lot* of disk space :-(

Some thoughts (and wild speculation) about scaling attention in LLMs, in my new post working through @sebastianraschka.com's "Build a Large Language Model (from Scratch)" www.gilesthomas.com/2025/05/llm-...

In my latest post on @sebastianraschka.com's "Build a Large Language Model (from Scratch)", I realise that attention heads are dumb -- and that's why they work so well. www.gilesthomas.com/2025/05/llm-...

What better use of a public holiday than learning how multi-head attention works? More notes from reading through @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...

Another attack of overthinking when going through the section on batches in @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...

"innit": a new system image for PythonAnywhere with support for Python up to version 3.13: blog.pythonanywhere.com/219/

Top people controlling a nuclear superpower.

Ever onwards! Now it's time to take a look at dropout, which was nice and simple until I started overthinking it :-) www.gilesthomas.com/2025/03/llm-...

About vibe coding, I got something to say. Most of people doing it can't code anyway: let them do what they want. In general: let people do the f**k they want. They will realize the strengths and limits of what they can do. They don't need babysitting (because you are *actually* scared).

Spent a couple of hours yesterday adding an /llms.txt parallel site to my blog. Thoughts welcome! www.gilesthomas.com/2025/03/llms...

After the trickiness of trainable self-attention, causal attention -- despite sounding complicated -- was a nice easy section :-) www.gilesthomas.com/2025/03/llm-...

Momentum regained! I was apprehensive about the workings of full self-attention, but like all of these things it made perfect sense after a bit of head-scratching: www.gilesthomas.com/2025/03/llm-...