Now this was a surprise. QKV bias is not meant to be useful -- but with my GPT-2 small model, it looks like it is! www.gilesthomas.com/2026/02/llm-...

Does removing dropout improve our baseline model's test loss? Yes, absolutely, and much more than gradient clipping did. www.gilesthomas.com/2026/02/llm-...

First "intervention" test: does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected. www.gilesthomas.com/2026/02/llm-...

Back to my LLM from scratch series. I want to train the *best* GPT-2-style model that I can locally in two days, and there are various levers to pull. Working out which ones work means I need a baseline for comparison. www.gilesthomas.com/2026/02/llm-...

I wanted to get a custom LLM up onto the @hf.co Hub, and couldn't find an in-depth tutorial. Here's the one I wish I'd found before I got started: www.gilesthomas.com/2026/01/cust...

I thought it would be good to share the base models I've been training on Hugging Face, and now they are :-) www.gilesthomas.com/2026/01/llm-...

I wanted to dig into why the results I got on instruction fine-tuning for each of my models didn't seem to match up well with the loss on the test set. Got some interesting results: www.gilesthomas.com/2026/01/2026...

Having trained a GPT-2 scale base model from scratch in 48 hours locally, I wanted to see if I could do the same faster and at a reasonable cost in the cloud. I could! www.gilesthomas.com/2026/01/llm-...

I managed to train my own base model from scratch on an RTX 3090! Very detailed notes here: www.gilesthomas.com/2025/12/llm-...

So, what's left to do in my series on building an LLM from scratch? And what follow-up series should I work on? Some musings: www.gilesthomas.com/2025/11/llm-...

The end of the beginning... running evals on our model using Llama 3 is the last part of the main body of @sebastianraschka.com's "Build an LLM (from scratch)". Here's my writeup: www.gilesthomas.com/2025/11/llm-...

Back on track with chapter 7 of "Build an LLM (from scratch)": notes on instruction fine-tuning of our GPT-2 model: www.gilesthomas.com/2025/10/llm-...

Back when I started messing with LLMs, it looked to me like you could get reasonably OK results for chat applications without instruction fine-tuning. So before getting into Chapter 7 of "Build an LLM (from scratch)", I decided to see if that was really true: www.gilesthomas.com/2025/10/llm-...

And the next step -- a code walkthrough of my PyTorch version of Karpathy's 2015-vintage RNNs. www.gilesthomas.com/2025/10/retr...

Chapter 6 was easy and fun! Fine-tuning an LLM for classification tasks, with some initially disappointing results -- but it all came out in the wash: www.gilesthomas.com/2025/10/llm-...

Part 22 is live: we finally train the LLM :-) Following @sebastianraschka.com's book, we train on Edith Wharton, then swap in GPT-2 (124M) weights for comparison. Notes on seeding, AdamW, temperature and top-k. www.gilesthomas.com/2025/10/llm-...

Decided to learn about some "retro" language models in parallel with LLMs; here's my first post on that -- Revisiting Karpathy’s 'The Unreasonable Effectiveness of Recurrent Neural Networks'. www.gilesthomas.com/2025/10/revi...

Next up in my LLM from scratch series, some serious yak shaving on something @sebastianraschka.com covers in a sidebar: perplexity. www.gilesthomas.com/2025/10/llm-...

Back to the main track of my LLM from scratch posts: cross entropy -- what it is and why we use it. www.gilesthomas.com/2025/10/llm-...

Part 3, and this was a fun one to write: How do LLMs work? From token IDs through to logits -- projections, matrix multiplications, and attention step-by-step. www.gilesthomas.com/2025/09/how-...

Part 2: "The maths you need to start understanding LLMs": www.gilesthomas.com/2025/09/math...

Part 1: "What AI chatbots are actually doing under the hood". www.gilesthomas.com/2025/08/what...

I wanted to do a "what I've learned so far" post to wrap up my notes on Chapter 4 of @sebastianraschka.com's "Build an LLM (from scratch)" but when it got to 6,000 words I suspected it was getting a bit long. So here's a review of upcoming attractions: www.gilesthomas.com/2025/08/llm-...

Now it's time to look into shortcut connections in @sebastianraschka.com's "Build an LLM (from scratch)" -- where the Talmud becomes a surprisingly useful metaphor! www.gilesthomas.com/2025/08/llm-...

Some thinking in public: if the FFNs in an LLM only work on one context vector at a time, why aren't they bitten by the fixed-length bottleneck? www.gilesthomas.com/2025/08/the-...

After a summer break, it's on to the feed-forward layer in @sebastianraschka.com's "Build an LLM (from scratch)" -- in which I discover that attention is not all you need. www.gilesthomas.com/2025/08/llm-...

The next step in my slow-but-steady progress through @sebastianraschka.com's "Build an LLM (from scratch)" -- what is layer normalisation, why do we do it, and how does it work? www.gilesthomas.com/2025/07/llm-...

Connect your AI coding assistant directly to PythonAnywhere! It works with Claude, Copilot, Cursor, and any MCP-compatible tool. See github.com/pythonanywhe...

New blog post, just documenting a bit of cruft-removal: porting old Fabric3 code to modern Fabric: www.gilesthomas.com/2025/06/fabr...

After 14 years of leading @pythonanywhere.com -- 11 as an independent company and 3 as part of @anacondainc.bsky.social, it's time for me to take a much-needed break, and today was my last day. It's been an amazing ride and I'd like to thank everyone that made it possible.

Moving onwards with @sebastianraschka.com's "Build an LLM (from scratch)". The last step in an LLM, going from context vectors to next-word prediction, looks insanely simple. How can it possibly work? www.gilesthomas.com/2025/05/llm-...

First impressions of Claude 4: vibes seem fixed. 3.5 was likeable, 3.7 less so. So far this seems like a return to form.

We've got Streamlit working on PythonAnywhere with our experimental website system! help.pythonanywhere.com/pages/Stream... -- The bad news is that you need a paid account, Streamlit uses up a *lot* of disk space :-(

Some thoughts (and wild speculation) about scaling attention in LLMs, in my new post working through @sebastianraschka.com's "Build a Large Language Model (from Scratch)" www.gilesthomas.com/2025/05/llm-...

In my latest post on @sebastianraschka.com's "Build a Large Language Model (from Scratch)", I realise that attention heads are dumb -- and that's why they work so well. www.gilesthomas.com/2025/05/llm-...

What better use of a public holiday than learning how multi-head attention works? More notes from reading through @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...

Another attack of overthinking when going through the section on batches in @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...

"innit": a new system image for PythonAnywhere with support for Python up to version 3.13: blog.pythonanywhere.com/219/

Top people controlling a nuclear superpower.

Ever onwards! Now it's time to take a look at dropout, which was nice and simple until I started overthinking it :-) www.gilesthomas.com/2025/03/llm-...

About vibe coding, I got something to say. Most of people doing it can't code anyway: let them do what they want. In general: let people do the f**k they want. They will realize the strengths and limits of what they can do. They don't need babysitting (because you are *actually* scared).

Spent a couple of hours yesterday adding an /llms.txt parallel site to my blog. Thoughts welcome! www.gilesthomas.com/2025/03/llms...

After the trickiness of trainable self-attention, causal attention -- despite sounding complicated -- was a nice easy section :-) www.gilesthomas.com/2025/03/llm-...

Momentum regained! I was apprehensive about the workings of full self-attention, but like all of these things it made perfect sense after a bit of head-scratching: www.gilesthomas.com/2025/03/llm-...

What if Ukraine did a mineral deal with Europe

I don’t know if Google can top this, their finest work to date

After a surprise viral hit with my "write the tutorial you wished you'd found" post on HN, I felt it was worth doing a followup on why I think it's still worth blogging even if AIs slurp everything up and no-one reads it directly. HN link as an experiment: news.ycombinator.com/item?id=4316...

"Writing the tutorial I wish I'd found when I started learning" seems to be the best recipe for blog posts, at least for me.

Sometimes, you have to grind through the fundamentals again to make sure you're building new knowledge on a stable foundation. For me, the best way to do that is to try to write the beginner's tutorial I wish I'd had the first time around.