Sebastian Raschka+

How Claude's Text Watermarking Works

Link to the slides: https://sebastianraschka.com/pdf/slides/2026-08-18-claude-watermarking.pdf This video explains how Anthropic's text watermarking for Claude works. In particular, it explains how the next-token sampling process in LLMs work, and how this is modified to implement the watermarking technique. A thorough understanding of the watermarking is useful because it helps us understand the implementation shortcomings as well. 0:00 How Claude's Text Watermarking Works 5:38 How LLM Text Generation Works 11:39 How Next-Token Sampling Works 16:19 From Sampling to Watermarking 18:23 Random

Build A Reasoning Model (From Scratch), Page 198

In listing 6.5 on page 198 of Build a Reasoning Model (From Scratch), torch.manual_seed(0) should be torch.manual_seed(5). This correction is needed to reproduce the generated response and the corresponding log-probability outputs later in Chapter 6. Future printings will include the correction.

What I Learned From Implementing LLM Architectures From Scratch (And How to Get Started)

LLM Architecture Gallery: https://llm-gallery.com In this talk, I discuss what we can learn from implementing LLM architectures from scratch in Python and PyTorch. The main idea is that to really understand how modern LLMs work, it helps to inspect the actual implementation details: attention variants, normalization layers, configuration files, KV cache optimizations, and the small architectural choices that often make a model work correctly. I also walk through how I approach new open-weight models, how I compare them against reference implementations, and what broader architecture trends

A Visual Tour of Modern LLM Architectures

LLM Architecture Gallery: https://sebastianraschka.com/llm-architecture-gallery/ In this video, I take you on a visual tour of modern LLM architectures and walk through the key ideas behind models like DeepSeek, Qwen3-Next, Kimi, Sarvam, Ling 2.5, and Nemotron. We look at what actually changed in recent LLM design, including grouped-query attention (GQA), sliding-window attention, multi-head latent attention (MLA), DeepSeek sparse attention, and hybrid linear attention. The goal of the gallery is to make it easier to compare architectures side by side, connect the diagrams back to papers, c

LLM Building Blocks & Transformer Alternatives

Resources: - Understanding and Coding the KV Cache in LLMs from Scratch: https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms - The Big Architecture Comparison: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison - Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers - Reasoning From Scratch book: https://mng.bz/Nwr7 Description: Learn the core components of modern transformer-based large language models (LLMs) and the practical techniques that make inference faster and cheaper. We walk throu

The Big LLM Architecture Comparison

Article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison Reasoning from scratch book: https://mng.bz/Nwr7 LLMs from Scratch repo: https://github.com/rasbt/LLMs-from-scratch This video covers the most important open-weight LLM architectures released in 2025, along with their architectural design decisions. 00:00:00 The Big Architecture Comparison 00:01:52 1. DeepSeek V3/R1 00:24:50 2. OLMo 2 00:35:07 3. Gemma 3 00:44:31 4. Mistral Small 3.1 00:48:04 5. Llama 4 00:50:03 6. Qwen3 00:58:07 7. SmolLM3 01:05:03 8. Kimi 2 01:08:19 9. GPT-OSS 01:14:57 10. Grok 2.5 01:19:

Build an LLM from Scratch 7: Instruction Finetuning

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to instruction finetune an LLM. 00:00 7.2 Preparing a dataset for supervised instruction finetuning 15:37 7.3 Organizing data into training batches 39:17 7.4 Creating data loaders for an instruction dataset 46:44 7.5 Loading a pretrained LLM 54:25 7.6 Finetuning the LLM on instruction data 1:14:20 7.7 Extracting and saving responses 1:23:56 7.8 Evaluating the finetuned LLM You can find a

Build an LLM from Scratch 6: Finetuning for Classification

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to finetune an LLM as a classifier (here using a spam classification example) as a gentle introduction to fine-tuning, before instruction finetuning the LLM in the next video. 00:00 6.2 Preparing the dataset 26:49 6.3 Creating data loaders 42:50 6.4 Initializing a model with pretrained weights 52:56 6.5 Adding a classification head 1:08:28 6.6 Calculating the classification loss and accu

Build an LLM from Scratch 5: Pretraining on Unlabeled Data

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This video explains how to pretrain a LLM from scratch. 00:00 5.1.1 Using GPT to generate text 17:23 5.1.2 Calculating the text generation loss: cross-entropy and perplexity 44:43 5.1.3 Calculating the training and validation set losses 1:10:27 5.2 Training an LLM 1:36:36 5.3 Decoding strategies to control randomness 1:39:51 5.3.1 Temperature scaling 1:53:24 5.3.2 Top-k sampling 2:01:40 5.3.3 Modifying the text generation function 2

Build an LLM from Scratch 4: Implementing a GPT model from Scratch To Generate Text

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to code an LLM architecture from scratch. 00:00 4.1 Coding an LLM architecture 13:52 4.2 Normalizing activations withlayer normalization 36:02 4.3 Implementing a feed forward network with GELU activations 52:16 4.4 Adding shortcut connections 1:03:18 4.5 Connecting attention and linear layers in a transformer block 1:15:13 4.6 Coding the GPT model You can find additional bonus materials

Build an LLM from Scratch 3: Coding attention mechanisms

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how attention mechanisms (self-attention, causal attention, multi-head attention) work by coding them from scratch. 00:00 3.3.1 A simple self-attention mechanism without trainable weights 41:01 3.3.2 Computing attention weights for all input tokens 52:40 3.4.1 Computing the attention weights step by step 1:12:33 3.4.2 Implementing a compact SelfAttention class 1:21:00 3.5.1 Applying a causal

Build an LLM from Scratch 2: Working with text data

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video going over text data preparations steps (tokenization, byte pair encoding, data loaders, etc.) for LLM training. 00:00 2.2 Tokenizing text 14:02 2.3 Converting tokens into token IDs 23:56 2.4 Adding special context tokens 30:26 2.5 Byte pair encoding 44:00 2.6 Data sampling with a sliding window 1:07:10 2.7 Creating token embeddings 1:15:45 2.8 Encoding word positions You can find additional bonus mate

Build an LLM from Scratch 1: Set up your code environment

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to set up a Python environment using uv. 00:00 Introduction 01:33 Setup info on GitHub 03:30 Optional setup preferences and uv 05:00 uv pip vs uv add syntax 05:35 1) Installing Python 09:05 2) Setting up uv 10:12 3) Creating a virtual environment 14:03 4) Installing packages 16:34 5) pip install fallback 16:57 6) If nothing works: Google Colab :) 19:07 7) uv run to run Jupyter Lab locall

Reinforcement Learning with Human Feedback (RLHF) in 4 minutes

Understanding Reinforcement Learning with Human Feedback (RLHF) – A short clip from my talk at the 2023 Optimized AI Conference (https://www.oaiconference.com/). Unfortunately, I won’t be attending in 2025 due to a scheduling conflict, but I highly recommend checking it out! If you want to read more about RLHF, here are some of my articles: 📌 LLM Training: RLHF and Its Alternatives → https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives 📌 Tips for LLM Pretraining & Evaluating Reward Models → https://magazine.sebastianraschka.com/p/tips-for-llm-pretraining-and-evalu

LLMs: A Journey Through Time and Architecture

REFERENCES: - Step-by-step guide converting GPT to Llama: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb - Build a Large Language Model (From Scratch): http://mng.bz/M96o - LitGPT: https://github.com/Lightning-AI/litgpt - The Llama 3 Herd of Models (31 July 2024), https://arxiv.org/abs/2407.21783 - Qwen2 Technical Report (15 July 2024), https://arxiv.org/abs/2407.10671 - Apple Intelligence Foundation Language Models (29 July 2024), https://arxiv.org/abs/2407.21075 - Gemma 2: Improving Open Language Models at a Practical Size (31 July 20

Building LLMs from the Ground Up: A 3-hour Coding Workshop

REFERENCES: 1. Build an LLM from Scratch book: https://amzn.to/4fqvn0D 2. Build an LLM from Scratch repo: https://github.com/rasbt/LLMs-from-scratch 3. GitHub repository with workshop code: https://github.com/rasbt/LLM-workshop-2024 4. Lightning Studio for this workshop: https://lightning.ai/lightning-ai/studios/llms-from-the-ground-up-workshop?view=public 5. LitGPT: https://github.com/Lightning-AI/litgpt DESCRIPTION: This tutorial is aimed at coders interested in understanding the building blocks of large language models (LLMs), how LLMs work, and how to code them from the ground up in PyTor