My "The Building Blocks of Today’s and Tomorrow’s Language Models" talk at the PyTorch Conference is now up on YouTube! youtube.com/watch?v=nDl6... The silver lining of my late arrival and rescheduling: There was no talk after mine, it's followed by a 30 min Q&A instead of just the usual 5 :)

I just saw the Kimi K2 Thinking release! Kimi K2 is based on the DeepSeek V3/R1 architecture, and here's a side-by-side comparison. In short, Kimi K2 is a slightly scaled DeepSeek V3/R1. And the gains are in the data and training recipes. Hopefully, we will see some details on those soon, too.

My new field guide to alternatives to standard LLMs: Gated DeltaNet hybrids (Qwen3-Next, Kimi Linear), text diffusion, code world models, and small reasoning transformers. 🔗 magazine.sebastianraschka.com/p/beyond-sta...

Just saw the benchmarks of the new open-weight MiniMax-M2 LLM, and the performance is too good to ignore :). So, I just amended my "The Big LLM Architecture Comparison" with entry number 13! Link to the full article: magazine.sebastianraschka.com/p/the-big-ll...

A short talk on the main architecture components of LLMs this year + a look beyond the transformer architecture: www.youtube.com/watch?v=lONy...

🔗 Mixture of Experts (MoE): github.com/rasbt/LLMs-f...

Chapter 3, and with it the first 176 pages, is now live! (mng.bz/lZ5B)

Sliding Window Attention 🔗 github.com/rasbt/LLMs-f...

Multi-Head Latent Attention 🔗 github.com/rasbt/LLMs-f...

Just a bit of weekend coding fun: A memory estimator to calculate the savings when using grouped-query attention vs multi-head attention (+ code implementations of course). 🔗 github.com/rasbt/LLMs-f... Will add this for multi-head latent, sliding, and sparse attention as well.

Updated & turned my Big LLM Architecture Comparison article into a video lecture. The 11 LLM archs covered in this video: 1. DeepSeek V3/R1 2. OLMo 2 3. Gemma 3 4. Mistral Small 3.1 5. Llama 4 6. Qwen3 7. SmolLM3 8. Kimi 2 9. GPT-OSS 10. Grok 2.5 11. GLM-4.5/4.6 www.youtube.com/watch?v=rNlU...

From the Hierarchical Reasoning Model (HRM) to a new Tiny Recursive Model (TRM). A few months ago, the HRM made big waves in the AI research community as it showed really good performance on the ARC challenge despite its small 27M size. (That's about 22x smaller than the smallest Qwen3 0.6B model.)

It only took 13 years, but dark mode is finally here sebastianraschka.com/blog/2021/dl...

How do we evaluate LLMs? I wrote up a new article on (1) multiple-choice benchmarks, (2) verifiers, (3) leaderboards, and (4) LLM judges All with from-scratch code examples, of course! sebastianraschka.com/blog/2025/ll...

Just shared a new article on "The State of Reinforcement Learning for LLM Reasoning"! If you are new to reinforcement learning, this article has a generous intro section (PPO, GRPO, etc) Also, I cover 15 recent articles focused on RL & Reasoning. 🔗 magazine.sebastianraschka.com/p/the-state-...

Coded Llama 3.2 model from scratch and shared it on the HF Hub. Why? Because I think 1B & 3B models are great for experimentation, and I wanted to share a clean, readable implementation for learning and research: huggingface.co/rasbt/llama-...

My next tutorial on pretraining an LLM from scratch is now out. It starts with a step-by-step walkthrough of understanding, calculating, and optimizing the loss. After training, we update the text generation function with temperature scaling and top-k sampling: www.youtube.com/watch?v=Zar2...

I'm right now on the last chapter of "Build a Large Language Model (from scratch)" by @sebastianraschka.com and it's absolutely amazing to get started. Now I can understand why people lose their shit over DeepSeek, for example

I just shared a new tutorial: Implementing GPT From Scratch! In this 1:45 h hands-on coding session, I go over implementing the GPT architecture, the foundation of modern LLMs (and I also have bonus material converting it to Llama 3.2): www.youtube.com/watch?v=YSAk...

Yesterday, Google released Gemma 3, their latest open-weight LLM. Finally, a new addition to the "Big 5" of open-weight models (Gemma, Llama, DeepSeek, Qwen, and Mistral). I just went through the Gemma 3 report and experimented a bit with the models, and there are plenty of interesting tidbits:

Just read Gemma 3 is out. Gemma models are super underestimated imho. Will be taking this for a spin in the next few days. In the meantime, they have a technical report here: storage.googleapis.com/deepmind-med...

Just uploaded my "Coding Attention Mechanisms" tutorial. A 2h15m session on coding attention mechanisms to understand how the engine of LLMs works: self-attention → parameterized self-attention → causal self-attention → multi-head self-attention www.youtube.com/watch?v=-Ll8...

I just shared a new article, "The State of Reasoning Models", where I am exploring 12 new research articles on improving the reasoning capabilities of LLMs (all published after the release of DeepSeek R1): magazine.sebastianraschka.com/p/state-of-l... Happy reading!

Takeaways from the latest State of ML Competitions report mlcontests.com/state-of-mac...: - Python & PyTorch still dominate - 80%+ use NVIDIA GPUs, but no multi-node setups 🤔 - LoRA still popular for training efficiency, but full finetuning gains traction. Surprisingly, CNNs still lead in CV comps

@gilesthomas.com is building an LLM from scratch as a Learning exercise and I am so jealous. Working off the book _Build a Large Language Model (from Scratch)_ by Sebastian Raschka First post in series here: www.gilesthomas.com/2024/12/llm-...

A new tutorial in my “Build A Large Language Model From Scratch” series is now live (www.youtube.com/watch?v=341R...) - Tokenizing raw text and converting tokens into token IDs - Applying byte pair encoding - Setting up data loaders in PyTorch for efficient training

Setting up your code environment is often the 1st step when building LLMs for education, research, or production. Made a new video where share my personal setup: "Build an LLM from Scratch 1: Set up your code environment": youtube.com/watch?v=yAcW... (Are videos like this helpful?)

Here’s the 2025 LLM roadmap 😊 1. Code and train your own LLM to really understand the fundamentals 2. Train models more conveniently using production-ready libraries 3. Learn about the big-picture considerations for real-world LLM/AI apps

It's 2025, and I’ve finally updated my Python setup guide to use uv + venv instead of conda + pip! Here's my go-to recommendation for uv + venv in Python projects for faster installs, better dependency management: github.com/rasbt/LLMs-f... (Any additional suggestions?)

Can we merge the query and key weight matrices in an LLM into a single covariance matrix and still train effectively? Here are some promising early results from a reader: github.com/rasbt/LLMs-f... Anyone else familiar with projects that tried this?

Maybe a hot take, but what about the following advice to the next gen: Don't get an AI degree; the curriculum will be outdated before you graduate. Instead, study math, stats, or physics as your foundation, and stay current with AI through code-focused books, blogs, and papers.

quite. btw found @sebastianraschka.com 's simple way refreshing in that regard: "when we say language models 'understand,' we mean that they can process and generate text in ways that appear coherent and contextually relevant, not that they possess human-like consciousness or comprehension"

Just read the s1: Simple Test-Time Scaling paper. Super interesting approach to improving reasoning models! TL;DR: 1. SFT on 1k curated examples w/ reasoning traces. 2. Control response length w/ budget forcing: "Wait" tokens → longer reasoning/self-correction. "Final Answer:" → enforce stopping.

I just finished writing up my take on reasoning models: magazine.sebastianraschka.com/p/understand... Here, I 1. Discuss the advantages & disadvantages of reasoning models 2. Of course, describe and discuss DeepSeek R1 3. Describe the 4 main ways to building & improving reasoning models

Remember when we were arguing about the term "foundation models"? It feels like ages ago! With those base models culminating in DeepSeek v3 in Dec 2024, 2025 will likely be the year of LLM specialization! Anyway, I will be writing about it more soon :).

But before I get to the reasoning model space... if you are looking to do some focused offline reading this weekend, I just re-compiled my take on the "noteworthy AI research papers of 2024" into one PDF-export-friendly 47-page mega-post with TOC and all: sebastianraschka.com/blog/2025/ll...

2024 was a great year for building general-purpose LLMs (with some special-purpose finetunes for math & code etc). Looks like 2025 is the year of diverging more into (1) reasoning models (for math and code) and (2) agents (for workflow automation). Exciting times!

🌟 Noteworthy AI Research Papers of 2024 👍🏼 These 2-part posts by @sebastianraschka.com discuss important AI research papers from 2024. Highly recommended! Part 1: https://buff.ly/40fMWfa Part 2: https://buff.ly/4g7F1p9 #AIresearch #GenAI

A reader recently shared a resource on training tokenizers for new languages which reminded me I originally wrote a BPE Tokenizer for my “LLMs from Scratch” book but never shared it! If you are looking a weekend project, here you go: sebastianraschka.com/blog/2025/bp...

Catching up on improving LLMs by scaling inference-time compute, new LLM scaling laws for precision, and more... "Noteworthy AI Research Papers 2024 Part 2" is now formatted and published. It was certainly an eventful year for this fast-moving field: magazine.sebastianraschka.com/p/ai-researc...

"Sky-T1-32B-Preview, our reasoning model that performs on par with o1-preview on popular reasoning and coding benchmarks." That was quick! Is this already the Alpaca moment for reasoning models? Source: novasky-ai.github.io/posts/sky-t1/

The model weights of Microsoft’s phi-4 14B LLM are now finally publicly available. Thanks to a kind community contribution, it’s now also available in LitGPT already (`litgpt finetune microsoft/phi-4`): github.com/Lightning-AI...

In case you were looking for the December 2024 LLM papers, I just added them to the list! magazine.sebastianraschka.com/i/152305086/...

Happy New Year! To kick off the year, I've finally been able to format and upload the draft of my AI Research Highlights of 2024 article. It covers a variety of topics, from mixture-of-experts models to new LLM scaling laws for precision:

It will take time, but as both a wish and a prediction for 2025, I hope to see more interesting approaches to developing open-weight (o1 and o3-like) reasoning models through improved post-training recipes, data, and inference compute.

Hey all, I've been a bit quiet the last couple of weeks as I am recovering from an accident & injury. Unfortunately, I couldn’t write my yearly AI research review this year, but here’s at least a list of bookmarked papers you might find useful: magazine.sebastianraschka.com/p/llm-resear...

If you find yourself with too much free time over the (long) weekend / holidays, I have ~3h Building an LLM from the Ground Up workshop on YouTube that may come in handy: m.youtube.com/watch?v=quh7...

Through the first 200 pages of "Build a Large Language Model" by Sebastian Raschka (@sebastianraschka.com). Bomber of a book. Very complete explanations with just the right amount of detail, even for a mechanical engineer like myself. Much recommend.

Outlined an AI research review article for December… I love traveling but I also can’t wait to be back on my computer 😅. In the meantime, if you are curious how Multimodal LLMs work, I recently wrote an article to explain the main & recent approaches: magazine.sebastianraschka.com/p/understand...

The weekends are for building LLMs. Join us at reading “Build a Large Language Model (from scratch)” by @sebastianraschka.com