Sebastian Raschka

ML researcher and educator; clear deep dives on LLM internals and from-scratch implementations.

Recommends

Books, films, shows, and podcasts they point to — tap a source to read the passage.

Books · 3

Build a Large Language Model from Scratch

Sebastian Raschka · 2024

Build an LLM from Scratch 3: Coding attention mechanismsSebastian Raschka Build an LLM from Scratch 4: Implementing a GPT model from Scratch To Generate TextSebastian Raschka

Context

Sebastian Raschka recommends this book as a structured, self-contained educational resource for building functional LLMs.

“so this is a supplementary coding a long video for the build a large language model from scratchbook”

→ Amazon

book

Build a Reasoning Model (From Scratch)

Build A Reasoning Model From Scratch 1: Motivation & Code SetupSebastian Raschka

Context

Sebastian Raschka values this book as the primary resource for readers transitioning from conventional LLMs to reasoning capabilities.

“This book essentially picks up where the other book left off.”

Build a Large Language Model (From Scratch)

Sebastian Raschka · 2024

Build A Reasoning Model From Scratch 1: Motivation & Code SetupSebastian Raschka

Context

Sebastian Raschka recommends this book to provide readers with the necessary foundational details on architecture, pre-training, and fine-tuning that support the reasoning model series.

“In this book, I covered all the essentials from coding the LLM architecture to pre-training and fine-tuning.”

→ Amazon

Sort

Sebastian Raschka10d

How LLM self-refinement works

Self-refinement asks an LLM to generate an answer, critique it, and use that critique to revise its response. Here, I walk through the process with a simple math example. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian Raschka13d

How majority voting works for LLMs

Generate several responses to the same prompt, then select the most frequent final answer. I walk through a simple example and explain why this is easier for numeric answers than for free-form text. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian Raschka14d

Build A Reasoning Model From Scratch 6: Reinforcement Learning 1 (Implementing GRPO for RLVR)

This video covers a from-scratch implementation of Reinforcement Learning with Verifiable Rewards (RLVR) using the Group Relative Policy Optimization (GRPO) algorithm in Python to train a small reasoning model. - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://amzn.to/4aAKiFY - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - LLMs from Scratch book: https://amzn.to/4fqvn0D - LLMs from Scratch repo: https://github.com/rasbt/LLMs-from-scratch - LLMs from Scratch playlist: https://www.youtube.com/watch?v=yAcWnf

Sebastian Raschka15d

How top-p filters tokens

Top-p sampling keeps the most likely tokens until their cumulative probability reaches a chosen threshold. I walk through an example with p = 0.8, then renormalize the retained probabilities so they sum to one. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian Raschka17d

Where verifiable rewards for reasoning model training come from

A calculator gives us a reference answer to check an LLM’s response. In this example, a matching final answer receives a reward of 1, and a mismatch receives 0. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian Raschka19d

How tokenizers turn text into numbers

A tokenizer splits text into tokens and maps them to numeric IDs. I walk through this process with the Qwen3 tokenizer, then show how decoding reconstructs the original text. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian Raschka21d

Build A Reasoning Model From Scratch 5: Inference Scaling 2 (Logprob Scoring, Self-Refinement)

This second video on inference-time scaling, introduces different scoring functions to break ties in majority voting and also introduces self-refinement. - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://amzn.to/4aAKiFY - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - LLMs from Scratch book: https://amzn.to/4fqvn0D - LLMs from Scratch repo: https://github.com/rasbt/LLMs-from-scratch - LLMs from Scratch playlist: https://www.youtube.com/watch?v=yAcWnfsZhzo&list=PLTKMiZHVd_2IIEsoJrWACkIxLRdfMlw11&index=1 0

Sebastian Raschka22d

Why learn with "from-scratch" code?

Diagrams help us understand an idea. Implementing it in code makes the details explicit and gives us something we can run and test. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1

Sebastian Raschka24d

What is the relationship between LLMs, Reasoning Models, and Agent Harnesses?

The reasoning model is the engine, and the agent harness is the software around it. Using a go-kart analogy, I explain how they fit together and why understanding the underlying LLM still matters. An excerpt from my reasoning model series: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1 Article on coding an agent harness: https://magazine.sebastianraschka.com/p/components-of-a-coding-agent

Sebastian Raschka26d

Do Looped Transformers Hide Chain of Thought?

Looped transformers reuse the same layers for additional computation. In this clip, I discuss why this might lead to a more compact chain of thought, using a math-exam analogy. An excerpt from my longer video on looped transformers: https://www.youtube.com/watch?v=KT4n-z_4QJU

Sebastian Raschka28d

Build A Reasoning Model From Scratch 4: Inference Scaling 1 (Temperature, Top-p, Self-Consistency)

After motivating the use cases for inference-time scaling, this video first walks through temperature scaling, top-p filtering, and multinomial sampling to generate diverse answers with an LLM. Then, using these techniques, we implement self-consistency to improve LLM answer accuracy. - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://amzn.to/4aAKiFY - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - LLMs from Scratch book: https://amzn.to/4fqvn0D - LLMs from Scratch repo: https://github.com/rasbt/LLMs-from-s

Sebastian Raschka28d

Different Tokens, Different Compute in LLMs

In Mixture-of-Recursions, a learned router decides which tokens continue through additional passes of the same transformer layers. Some tokens take one pass, others two or three. An excerpt from my longer video on looped transformers: https://www.youtube.com/watch?v=KT4n-z_4QJU

Sebastian Raschka1mo

Build A Reasoning Model From Scratch 3: The Verifier for Evaluation and RL with Verifiable Rewards

This video explains how to implement a verifier so that we can evaluate the LLM on math benchmarks. And it's also the same verifier we will reuse for the reinforcement learning with verifiable rewards (RLVR) training later on. - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://amzn.to/4aAKiFY - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - Reasoning effort article: https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms - LLMs from Scratch book: https://amzn.to/4fqvn0D - LLMs from Scra

Sebastian Raschka1mo

Build A Reasoning Model From Scratch 2: Loading a Base Model, Text Generation, and KV Caching

The second video in the Build A Reasoning Model (From Scratch) series illustrates how to load a pre-trained model, how LLM text generation works, and how to make it more efficient with KV caching. (These are the basics to get started, later videos will revisit text-generation with sampling in more detail.) - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://mng.bz/Nwr7 (Manning) & https://amzn.to/4aAKiFY (Amazon) - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - Reasoning From Scratch Playlist: https://www.you

Sebastian Raschka1mo

OpenAI Astra and Recurrent Depth / Looped Transformers

A short video explaining the rumored recurrent depth (looped transformer) architectural design in OpenAI's new (upcoming) Astra model. In particular, I discuss what looped transformers are and go over the related literature. UPDATE: Expanding on the video, I also put together a longer write-up on the topic: https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and Other links and references The Information article on OpenAI Astra: https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns Looped depth sharing explainer: h

Sebastian Raschka1mo

Build A Reasoning Model From Scratch 1: Motivation & Code Setup

This video explains the relationship between conventional LLMs and reasoning models (and agents), philosophizes a about "from scratch" approaches, and explains how to install Python & PyTorch requirements with uv. - Reasoning playlist: https://www.youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs - Reasoning Book: https://amzn.to/4aAKiFY - Reasoning GitHub repo: https://github.com/rasbt/reasoning-from-scratch - Reasoning effort article: https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms - LLMs from Scratch book: https://amzn.to/4fqvn0D - LLMs from Scratch repo: htt

Sebastian Raschka1mo

How Claude's Text Watermarking Works

Link to transcript: https://magazine.sebastianraschka.com/p/claude-watermarking Link to the slides: https://sebastianraschka.com/pdf/slides/2026-08-18-claude-watermarking.pdf This video explains how Anthropic's text watermarking for Claude works. In particular, it explains how the next-token sampling process in LLMs work, and how this is modified to implement the watermarking technique. A thorough understanding of the watermarking is useful because it helps us understand the implementation shortcomings as well. 0:00 How Claude's Text Watermarking Works 5:38 How LLM Text Generation Works 11:3

Sebastian Raschka3mo

Build A Reasoning Model (From Scratch), Page 198

In listing 6.5 on page 198 of Build a Reasoning Model (From Scratch), torch.manual_seed(0) should be torch.manual_seed(5). This correction is needed to reproduce the generated response and the corresponding log-probability outputs later in Chapter 6. Future printings will include the correction.

Sebastian Raschka5mo

What I Learned From Implementing LLM Architectures From Scratch (And How to Get Started)

LLM Architecture Gallery: https://llm-gallery.com In this talk, I discuss what we can learn from implementing LLM architectures from scratch in Python and PyTorch. The main idea is that to really understand how modern LLMs work, it helps to inspect the actual implementation details: attention variants, normalization layers, configuration files, KV cache optimizations, and the small architectural choices that often make a model work correctly. I also walk through how I approach new open-weight models, how I compare them against reference implementations, and what broader architecture trends

Sebastian Raschka6mo

A Visual Tour of Modern LLM Architectures

LLM Architecture Gallery: https://sebastianraschka.com/llm-architecture-gallery/ In this video, I take you on a visual tour of modern LLM architectures and walk through the key ideas behind models like DeepSeek, Qwen3-Next, Kimi, Sarvam, Ling 2.5, and Nemotron. We look at what actually changed in recent LLM design, including grouped-query attention (GQA), sliding-window attention, multi-head latent attention (MLA), DeepSeek sparse attention, and hybrid linear attention. The goal of the gallery is to make it easier to compare architectures side by side, connect the diagrams back to papers, c

Sebastian Raschka11mo

LLM Building Blocks & Transformer Alternatives

Resources: - Understanding and Coding the KV Cache in LLMs from Scratch: https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms - The Big Architecture Comparison: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison - Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers - Reasoning From Scratch book: https://mng.bz/Nwr7 Description: Learn the core components of modern transformer-based large language models (LLMs) and the practical techniques that make inference faster and cheaper. We walk throu

Sebastian Raschka13mo

The Big LLM Architecture Comparison

Article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison Reasoning from scratch book: https://mng.bz/Nwr7 LLMs from Scratch repo: https://github.com/rasbt/LLMs-from-scratch This video covers the most important open-weight LLM architectures released in 2025, along with their architectural design decisions. 00:00:00 The Big Architecture Comparison 00:01:52 1. DeepSeek V3/R1 00:24:50 2. OLMo 2 00:35:07 3. Gemma 3 00:44:31 4. Mistral Small 3.1 00:48:04 5. Llama 4 00:50:03 6. Qwen3 00:58:07 7. SmolLM3 01:05:03 8. Kimi 2 01:08:19 9. GPT-OSS 01:14:57 10. Grok 2.5 01:19:

Sebastian Raschka18mo

Build an LLM from Scratch 7: Instruction Finetuning

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to instruction finetune an LLM. 00:00 7.2 Preparing a dataset for supervised instruction finetuning 15:37 7.3 Organizing data into training batches 39:17 7.4 Creating data loaders for an instruction dataset 46:44 7.5 Loading a pretrained LLM 54:25 7.6 Finetuning the LLM on instruction data 1:14:20 7.7 Extracting and saving responses 1:23:56 7.8 Evaluating the finetuned LLM You can find a

Sebastian Raschka18mo

Build an LLM from Scratch 6: Finetuning for Classification

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to finetune an LLM as a classifier (here using a spam classification example) as a gentle introduction to fine-tuning, before instruction finetuning the LLM in the next video. 00:00 6.2 Preparing the dataset 26:49 6.3 Creating data loaders 42:50 6.4 Initializing a model with pretrained weights 52:56 6.5 Adding a classification head 1:08:28 6.6 Calculating the classification loss and accu

Sebastian Raschka19mo

Build an LLM from Scratch 5: Pretraining on Unlabeled Data

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This video explains how to pretrain a LLM from scratch. 00:00 5.1.1 Using GPT to generate text 17:23 5.1.2 Calculating the text generation loss: cross-entropy and perplexity 44:43 5.1.3 Calculating the training and validation set losses 1:10:27 5.2 Training an LLM 1:36:36 5.3 Decoding strategies to control randomness 1:39:51 5.3.1 Temperature scaling 1:53:24 5.3.2 Top-k sampling 2:01:40 5.3.3 Modifying the text generation function 2

Sebastian Raschka19mo

Build an LLM from Scratch 4: Implementing a GPT model from Scratch To Generate Text

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to code an LLM architecture from scratch. 00:00 4.1 Coding an LLM architecture 13:52 4.2 Normalizing activations withlayer normalization 36:02 4.3 Implementing a feed forward network with GELU activations 52:16 4.4 Adding shortcut connections 1:03:18 4.5 Connecting attention and linear layers in a transformer block 1:15:13 4.6 Coding the GPT model You can find additional bonus materials

Sebastian Raschka19mo

Build an LLM from Scratch 3: Coding attention mechanisms

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how attention mechanisms (self-attention, causal attention, multi-head attention) work by coding them from scratch. 00:00 3.3.1 A simple self-attention mechanism without trainable weights 41:01 3.3.2 Computing attention weights for all input tokens 52:40 3.4.1 Computing the attention weights step by step 1:12:33 3.4.2 Implementing a compact SelfAttention class 1:21:00 3.5.1 Applying a causal

Sebastian Raschka19mo

Build an LLM from Scratch 2: Working with text data

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video going over text data preparations steps (tokenization, byte pair encoding, data loaders, etc.) for LLM training. 00:00 2.2 Tokenizing text 14:02 2.3 Converting tokens into token IDs 23:56 2.4 Adding special context tokens 30:26 2.5 Byte pair encoding 44:00 2.6 Data sampling with a sliding window 1:07:10 2.7 Creating token embeddings 1:15:45 2.8 Encoding word positions You can find additional bonus mate

Sebastian Raschka19mo

Build an LLM from Scratch 1: Set up your code environment

Links to the book: - https://amzn.to/4fqvn0D (Amazon) - https://mng.bz/M96o (Manning) Link to the GitHub repository: https://github.com/rasbt/LLMs-from-scratch This is a supplementary video explaining how to set up a Python environment using uv. 00:00 Introduction 01:33 Setup info on GitHub 03:30 Optional setup preferences and uv 05:00 uv pip vs uv add syntax 05:35 1) Installing Python 09:05 2) Setting up uv 10:12 3) Creating a virtual environment 14:03 4) Installing packages 16:34 5) pip install fallback 16:57 6) If nothing works: Google Colab :) 19:07 7) uv run to run Jupyter Lab locall

Sebastian Raschka20mo

Reinforcement Learning with Human Feedback (RLHF) in 4 minutes

Understanding Reinforcement Learning with Human Feedback (RLHF) – A short clip from my talk at the 2023 Optimized AI Conference (https://www.oaiconference.com/). Unfortunately, I won’t be attending in 2025 due to a scheduling conflict, but I highly recommend checking it out! If you want to read more about RLHF, here are some of my articles: 📌 LLM Training: RLHF and Its Alternatives → https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives 📌 Tips for LLM Pretraining & Evaluating Reward Models → https://magazine.sebastianraschka.com/p/tips-for-llm-pretraining-and-evalu

Sebastian Raschka25mo

LLMs: A Journey Through Time and Architecture

REFERENCES: - Step-by-step guide converting GPT to Llama: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb - Build a Large Language Model (From Scratch): http://mng.bz/M96o - LitGPT: https://github.com/Lightning-AI/litgpt - The Llama 3 Herd of Models (31 July 2024), https://arxiv.org/abs/2407.21783 - Qwen2 Technical Report (15 July 2024), https://arxiv.org/abs/2407.10671 - Apple Intelligence Foundation Language Models (29 July 2024), https://arxiv.org/abs/2407.21075 - Gemma 2: Improving Open Language Models at a Practical Size (31 July 20

Sebastian Raschka25mo

Building LLMs from the Ground Up: A 3-hour Coding Workshop

REFERENCES: 1. Build an LLM from Scratch book: https://amzn.to/4fqvn0D 2. Build an LLM from Scratch repo: https://github.com/rasbt/LLMs-from-scratch 3. GitHub repository with workshop code: https://github.com/rasbt/LLM-workshop-2024 4. Lightning Studio for this workshop: https://lightning.ai/lightning-ai/studios/llms-from-the-ground-up-workshop?view=public 5. LitGPT: https://github.com/Lightning-AI/litgpt DESCRIPTION: This tutorial is aimed at coders interested in understanding the building blocks of large language models (LLMs), how LLMs work, and how to code them from the ground up in PyTor

No appearances yet. If you've seen Sebastian Raschka as a guest somewhere, hit + suggest to add it.

Similar thinkers

Minds whose work overlaps with Sebastian Raschka’s.

Shared niche — deep learning architecture scholarship

Sasha Rush

@srush

Cornell professor; clear technical deep dives on LLM architecture.

Open Profile

Shared niche — undefined intellectual scope

Christopher Manning

@christopher-manning

Stanford NLP director; foundational textbooks and research on language models.

Open Profile

Shared niche — applied machine learning engineering

Eugene Yan

@eugeneyan

Applied scientist; essays on AI engineering, evaluation, and recommender systems.

Open Profile

Shared niche — foundational machine learning research

Yann LeCun

@yann-lecun

Chief AI Scientist at Meta; pioneer of convolutional neural networks and deep learning.

Open Profile

Shared niche — visual pedagogical deep learning

Jay Alammar

@jalammar

Visual explanations of transformers, attention, and embeddings.

Open Profile

Shared niche — critical ai behavior analysis

Janelle Shane

@janelle-shane

Optics researcher; AI Weirdness blog on the absurdities of generative models.

Open Profile

Shared niche — model interpretability and research

Sara Hooker

@sararosehooker

Cohere For AI head; on equitable ML systems and data-centric research.

Open Profile

Shared niche — technical deep learning synthesis

Lilian Weng

@lilianweng

Research engineer; rigorous long-form posts on LLM agents, RLHF, and diffusion models.

Open Profile

Is this you? Claim this page →

Sebastian Raschka

Recommends

Channel content

Appearances

Similar thinkers

Recommends

Channel content

Appearances

Similar thinkers