Chapter 1 closed with a budget calculator and a warning: Chinchilla is compute-optimal for training, not for deployment, so labs deliberately over-train small models on more tokens than the ratio says. This chapter is the other axis. Same compute, same token count - where does it actually go once it lands on the cluster? We walk the six places a modern flagship has changed since GPT-3: the data (crawl to classifier to synthetic), the block (norms, activation, positions), attention (a KV-cache story that reshapes both training and inference), sparsity (MoE and multi-token prediction), numerics (BF16 to FP8 with a new optimizer), and the tail (mid-training, distillation, the one non-attention alternative that survived). By the end, the machine looks modern - but the model it produces is still a document completer. That gap opens Chapter 3.
1. What a flagship run is actually buying
A 2020-era pretraining run was a small number of moving parts: a dense transformer, BPE tokens, AdamW, FP16 or BF16, next-token cross-entropy, one big pile of scraped text. A 2025-26 flagship run is a much longer bill. The DeepSeek-V3 technical report is the clearest single window into it: 671B total parameters, 37B active per token (a mixture-of-experts), multi-head latent attention to shrink the KV cache, auxiliary-loss-free load balancing on the router, multi-token prediction heads on top, FP8 mixed-precision training throughout, 14.8T pretraining tokens - and pretraining alone cost 2.664 million H800 GPU-hours (2.788M total including long-context extension and post-training). Every one of those choices is a lever you did not have five years ago.
The point of the chapter is not to catalog names. It is to see the shape of the machine a working practitioner reasons about in 2026 - because every downstream chapter (post-training, reasoning, inference, agents) inherits this substrate. Tokenization, the seventh moving part, belongs one chapter back: Chapter 0 now teaches BPE, byte-level BPE, and the vocab creep from 32k up to 128k-200k, so we do not re-teach it here.
Data (what the model reads) · Block (RMSNorm, SwiGLU, RoPE) · Attention (MQA / GQA / MLA + FlashAttention) · Sparsity (MoE + MTP) · Numerics (BF16, FP8, Muon) · Tail (long-context extension, mid-training, distillation). Sections 2-8 walk them in that order.
2. The data pipeline is the model
Chinchilla put a number on how many tokens. It did not say which ones. The post-Chinchilla frontier's largest single quality lever has been the pipeline that picks them.
The 2024 pipelines converged on the same skeleton. Start from Common Crawl. Strip HTML and boilerplate. Apply language ID. Deduplicate globally (MinHash / suffix-array, within-snapshot and across snapshots). Filter with a model-based quality classifier that scores each document. Decontaminate against known eval sets. Blend in synthetic text and code/math sub-corpora. Anneal on the best of it at the end.
Three 2024 releases are the reference points for the classifier step:
| Dataset | Filter signal | Scale | Trick |
|---|---|---|---|
| DCLM | fastText classifier trained on OpenHermes + ELI5 | 3.8T tokens filtered from 240T | Whole training a benchmark for filters |
| FineWeb-Edu | Classifier on educational ratings from Llama-3-70B | 1.3T tokens | Small model teaches the filter for a big one |
| Nemotron-CC | Classifier ensemble + LLM rephrasing of low-quality docs | 4.4T real + 1.9T synthetic = 6.3T | Rewriting recovers ~90% of filtered content |
The Nemotron trick generalizes: synthetic data is now first-class. The most public line of that argument is Microsoft's phi series - phi-1 (2023) trained on "textbooks are all you need" - and a 1B model that beat much larger contemporaries on code. The result was not that synthetic beats real. It was that a distribution shaped by what a strong teacher would say next was worth more per token than raw crawl. Frontier runs now mix in curated math (proofs, solved problems), code (repo-scale, unit-test-verified), instruction-shaped documents, and machine-rewritten crawl.
The last data-side stage is the one everyone forgot to name until 2024: mid-training, also called cooldown or annealing. Save your very-highest-quality tokens for the tail of pretraining - math, code, textbooks, distilled reasoning traces - and warm the learning rate down while feeding only that. OLMo 2's "Dolmino" mix (Groeneveld et al. 2025) is the cleanest public case study; the technique also shows up under names like "cooldown" or "annealing" in the Llama 3 herd (Grattafiori et al. 2024). Chapter 3 will do the same trick with human-labeled preferences; the shape is the same, only the labels differ.
The C4-era pipelines were dominated by hand-written heuristics: bad-word blocklists, document length windows, stop-word ratios. That worked until the crawl grew. A model-based classifier can see coherence and pedagogy in a document; a regex sees neither. The gain is measurable and cheap: FineWeb-Edu pushed MMLU on a fixed-scale probe from 33% to 37% by filtering alone. Data quality is now the axis with the best return-on-effort in a frontier run, which is why every new lab publishes its filter before it publishes its model.
3. The block, modernized
Chapter 0 walked you through the 2017 block: multi-head self-attention, position-wise feed-forward, LayerNorm, sinusoidal positions, ReLU/GELU somewhere in the middle. That block is not what a 2025 flagship trains. Three swaps stuck, and they are boring in the good way - each one gives up a small amount of expressivity for a lot of stability, throughput, or generalization.
| Component | 2017 default | 2025 default | Why |
|---|---|---|---|
| Norm placement | Post-norm | Pre-norm | Stable gradients at 100+ layers |
| Norm | LayerNorm | RMSNorm | Same behavior, fewer ops, no mean subtraction |
| FFN activation | ReLU / GELU | SwiGLU | Gated variant, small loss win at fixed FLOPs |
| Positions | Sinusoidal / learned | RoPE | Rotary, relative, extensible past training length |
The whole story is on the Llama page. Touvron et al. (2023, arXiv:2302.13971) write, verbatim, "compared to the original architecture, here are the main differences [...]: Pre-normalization [GPT-3]. Instead of normalizing the output of each transformer sub-layer, we normalize the input, using the RMSNorm normalizing function [Zhang and Sennrich, 2019]. SwiGLU activation function [PaLM]. We replace the ReLU non-linearity by the SwiGLU activation function [...] introduced by Shazeer (2020). Rotary Embeddings [GPTNeo]. We remove the absolute positional embeddings and instead add rotary positional embeddings (RoPE), introduced by Su et al. (2021)." That short list is the modern block. Every open flagship since - Llama 2/3, Mistral, Qwen, DeepSeek, Gemma, Kimi - inherits the three swaps.
Two things are worth pausing on. RoPE (Su et al. 2021) encodes position by rotating each query and key vector by an angle that scales with token index and dimension frequency; the dot product then depends only on the relative offset. That relativity is what makes context extension possible at all (Section 5). And SwiGLU is a gated variant that multiplies two projections instead of one - three matrices in the FFN instead of two - which is why frontier reports tend to quote FFN hidden width as 2/3 of the naive figure to keep the parameter count honest.
4. Attention and the KV cache
At training time, standard multi-head attention costs $O(L^2 d)$ per layer. That has not changed. What has changed is the cost people design against, which is not training time - it is inference memory bandwidth. During autoregressive generation the model reads its own KV cache once per token; the cache is per-layer, per-head, per-token, and it grows with context length. At 128k context and dozens of layers a dense-MHA cache is measured in tens of gigabytes per sequence. That is the number the block is now optimized for.
Three shrinks lived long enough to matter:
- MQA (multi-query attention, Shazeer 2019): one K/V head shared by all Q heads. Divides the cache by number of heads. Loses some quality in ablation.
- GQA (grouped-query attention, Ainslie et al. 2023): groups of Q heads share one K/V head. A tunable knob between MHA and MQA. Llama 2 70B was the flagship that made it standard.
- MLA (multi-head latent attention, DeepSeek-V2 2024): do not cache K and V. Cache a low-rank latent, reconstruct per-head K and V on the fly with two up-projections. Shrinks the cache more than GQA and, per DeepSeek's ablations, keeps quality above MHA rather than below it.
The demo below runs the arithmetic on one reference shape - DeepSeek-V3's own (128 heads, head_dim 128, 61 layers, BF16). Drag the context slider; the four bars show per-sequence KV cache under MHA, GQA-8, MQA, and the actual MLA that DeepSeek trained. MHA / GQA / MQA are the counterfactual dense-attention caches on the SAME shape, so the numbers are apples-to-apples. This is the calculation a serving team runs before quoting a token-per-dollar number.
Orthogonal to what you cache is how you compute the attention. FlashAttention (Dao et al. 2022), extended by FlashAttention-2 and FlashAttention-3, is exact attention - same output as the naive matmul - but reordered to never materialize the $L \times L$ score matrix in high-bandwidth memory. The algebra tiles the softmax so the score matrix lives only in on-chip SRAM. The word to remember is IO-aware: it wins by moving fewer bytes, not by doing fewer FLOPs. Every modern trainer uses it. It is the reason "attention is expensive" has stopped being the right mental model at the sizes anyone actually runs.
DeepSeek-V2 and V3 pair MLA (cache shrink) with the MoE that dominates their parameter count. The reason is symmetric: the router activates a small fraction of experts per token, so per-token FLOPs are low; MLA drops per-token memory bandwidth proportionally; both together make the model serveable. Take one away and the other gains a large fraction of its benefit back as slack.
5. Long context, extended after the fact
A 128k or 1M context window is almost never native. It is a fine-tune on top of a model that pretrained at 4k or 8k. The pipeline for extending it has two moves.
First, rescale the RoPE frequencies. Naive position interpolation (Chen et al. 2023) scales all rotary angles by the same ratio so that a longer sequence "looks like" the training length. That works but degrades short-context perplexity. YaRN (Peng et al. 2023) does the interpolation per frequency band - leave high-frequency dimensions alone, stretch low-frequency ones - and adds an attention-temperature term to keep the softmax well-scaled. YaRN is what most public 128k-context fine-tunes ride on; DeepSeek-V3 uses it to reach 128k, and the Llama 3.1 herd extends to 128k with a related recipe.
Second, run a dedicated long-context stage at the end of pretraining on a curriculum that grows the sequence length in bands (say 8k → 32k → 128k), with a data mixture reweighted toward long documents (books, code repos, legal filings) and needle-in-a-haystack style probes to sanity-check retrieval. This stage is small compared to main pretraining but has to be there. Without it, the model has RoPE angles it has never seen fire in a real gradient.
6. Sparsity: mixture-of-experts
By 2024 the frontier's open flagships were nearly all MoE: Mixtral (8 experts, top-2), DeepSeekMoE, DeepSeek-V3 (256 routed experts + 1 shared, 8-of-256 active, 671B total / 37B active), Kimi K2 (1T total / 32B active), gpt-oss (Aug 2025 open-weights release). The reason is arithmetic. If every token routes to $k$ of $n$ experts, the model behaves at inference like a $k/n$-of-total dense model for FLOPs, while every expert can specialize in the way a dense FFN cannot.
Replace the position-wise FFN in a transformer block with $N$ parallel FFNs (the experts) and a small learned gate. For each token, the gate scores all experts, top-$k$ are chosen ($k$ is small - 2 or 8), the token is routed to those experts, and their outputs are combined by gate weight. Total parameters = $\Theta(N)$, active parameters per token = $\Theta(k)$. Only the active experts do work, so both training FLOPs per token and inference FLOPs per token stay near the dense-$k$-expert cost.
Two 2024 refinements are what make current MoE runs actually work.
Fine-grained + shared experts (DeepSeekMoE 2024): instead of a handful of large experts, use many small ones (finer specialization) and keep one or two "shared" experts that every token visits (common knowledge that would otherwise be replicated). This is why DeepSeek-V3 has 256 routed experts and 1 shared.
Auxiliary-loss-free load balancing (DeepSeek-V3, following Wang et al. 2024): the classical MoE trick was to add a load-balancing loss - a penalty that pushes the router toward uniform expert usage. It works, but the penalty leaks into the language loss and slightly hurts quality. Aux-loss-free balancing instead maintains a per-expert bias that is adjusted online: if an expert is underused this step, nudge its bias up so it wins slightly more routing decisions; if overused, nudge it down. No loss term, no gradient distortion, same balance. DeepSeek reports this alone was worth a measurable quality bump.
The demo shows the router in miniature: eight routed experts, one shared, top-2 gating, fixed seed, a small stream of tokens colored by which experts fired. Watch total parameters stay pinned while active parameters per token drop, and watch the balancing bias keep any one expert from monopolizing traffic.
6.1 Multi-token prediction (MTP)
A quieter DeepSeek-V3 innovation. Instead of a single next-token head, train several shallow heads that predict the next $D$ tokens in parallel (each conditioned on the previous head's prediction). The training loss becomes a sum over depth, and each token contributes gradients from $D$ prediction sites instead of one. The paper reports the quality gain and an under-appreciated bonus: the extra heads work as draft-token producers for speculative decoding at inference time, so an MTP-trained model gets a serving speedup for free. Chapter 8 will pick this up again from the inference side.
7. Numerics and the optimizer
Pretraining precision has descended a step. BF16 (brain-float 16) replaced FP16 as the default around 2020 because it preserved the FP32 exponent range and stopped needing loss scaling. Then, in December 2024, DeepSeek-V3 shipped a production FP8 training path: matmuls in FP8 (E4M3 / E5M2) on Hopper tensor cores, with BF16 master weights and per-tile activation scaling to contain the numeric drift. The compute win is roughly 2× over BF16 at the same hardware. The engineering cost is real - overflow behavior, gradient scaling, kernel rewrites - but the FP8 recipe now lives in open trainers and other flagships (Nemotron, Llama post-training) have followed.
Optimizers moved less, and then a lot. AdamW was the frontier default from ~2019 through 2024. In 2024 Moonshot published Muon, a matrix-aware optimizer that treats a layer's weight update as a matrix and applies a Newton-Schulz-based orthogonalization step to its momentum. It converges faster than AdamW at fixed compute in the settings where it works, and it is cheap because Newton-Schulz needs only a few matmuls. The catch was stability: at large scale, MoE runs would spike. Kimi K2's MuonClip (July 2025) added a "QK-clip" that rescales attention query/key projections whenever their logits blow up. Kimi K2 - 1T total, 32B active - was pretrained on 15.5T tokens with zero loss spikes, per the release. That is the first published trillion-parameter run to drop AdamW at frontier scale.
8. How it fits on a cluster
Distributed training is not a technique, it is a wardrobe of them, and the frontier uses several at once. The three axes to know:
| Kind | What it splits | What it costs |
|---|---|---|
| Data / FSDP (ZeRO) | Different microbatches per GPU; params/grads/optimizer state sharded across GPUs | All-gather per forward step |
| Tensor (TP) | Each matmul split across GPUs by rows/cols | All-reduce inside every block; needs fast NVLink |
| Pipeline (PP) | Different layers on different GPUs | Bubble time unless you interleave microbatches |
| Expert (EP) | Different MoE experts on different GPUs | All-to-all traffic per token routing decision |
Modern runs stack all four. The DeepSeek-V3 report spends much of its infrastructure section on custom kernels that overlap EP all-to-all with computation, which is what keeps a heavily expert-parallel MoE from spending most of its wall-clock on networking. Chapter 0's parallelism demo shows the primitive that makes any of this possible - transformer blocks compute across the sequence axis in one pass, unlike an RNN - and this section is the industrial extension of that primitive.
9. The tail: mid-training, distillation, the SSM aside
A frontier run ends messier than it starts. After main pretraining there is a bundle of stages that used to be lumped into "training" and are now separately named. Mid-training / annealing (Section 2, tail of the data story), long-context extension (Section 5), sometimes a "capabilities" stage that feeds concentrated math/code, and finally a bake-off between many small distilled variants.
Distillation as pretraining earned its own status with the Gemma line. Gemma 2 (Google DeepMind 2024) trained a 27B teacher from scratch and then trained the 2B and 9B students on the teacher's full soft distribution over the next token, not the one-hot label. Each token gives the small model a richer gradient - the entire probability vector, not "yes / no." Gemma 3 continued the recipe. This is not the reasoning distillation you will meet in Chapter 6 (that copies traces, not logits); this is pretraining-time logit distillation, and it is why small open-weights models have gotten as good as they have.
Finally, the one non-attention architecture that survived. Mamba (Gu & Dao 2023) and Mamba-2 are selective state-space models: instead of all-to-all attention, they run a linear-time recurrence with data-dependent transitions, keeping constant per-step state instead of a growing KV cache. Pure SSMs match small transformers on language and lose ground on retrieval-heavy tasks. What is shipping in 2025-26 is hybrid: alternating SSM layers and attention layers in the same stack. Nvidia's Nemotron-H (Nvidia 2025) and IBM/AI21's earlier Jamba are the reference points. The hybrid keeps attention where it earns its keep (in-context precision) and pays SSM cost everywhere else.
10. What you get, and what you do not
Add it up. Run for a few million H100/H800-hours. Feed 10-20T tokens through a modernized MoE transformer with MLA and RoPE, in FP8, with Muon, on a cluster that overlaps expert parallelism with matmul, extend the context to 128k, mid-train on your best tokens, distill down for the small sizes. The model that falls out is - still - a document completer.
Ask it "explain photosynthesis to a child" and, half the time, you get a plausible continuation of the prompt rather than an answer. Ask it to solve a word problem and it may or may not show its work. Ask it what its policy is on unsafe requests and it does not have one. Pretraining has taught it the substrate: what the world talks about, in what order, in what registers. It has not taught it that there is a user across the table who wants an answer to a specific question. That is the intent-alignment gap. Chapter 3 is where a small team at OpenAI closes it with a three-stage recipe called RLHF, ships the result as ChatGPT, and quietly plants the seed of chain-of-thought.
Chapter 0 turned the architecture lever (attention). Chapter 1 turned the scale lever (Kaplan, Chinchilla, over-train for deployment). Chapter 2 turned the efficiency lever - same objective, machine rebuilt from data pipeline through numerics. Chapter 3 turns the alignment lever. Same weights. Different behavior.
11. What to take with you
- Data is a pipeline, not a corpus. Crawl, dedup, classifier-filter, decontaminate, mix in synthetic and code/math, save the best tokens for a cooldown stage. DCLM, FineWeb-Edu, Nemotron-CC are the 2024 reference filters.
- The 2017 block quietly grew up. Pre-norm RMSNorm, SwiGLU FFN, RoPE positions. Every flagship since Llama 1 uses that triple.
- Attention now optimizes for inference memory. MQA / GQA / MLA all shrink the KV cache; FlashAttention makes exact attention IO-cheap. Long context is a rescale (YaRN) plus an extension stage.
- Sparsity is the flagship default. MoE separates total from active parameters. Fine-grained + shared experts, aux-loss-free balancing, and multi-token prediction are the 2024-25 refinements that made it stable.
- Numerics dropped a step. BF16 became FP8 for matmuls (DeepSeek-V3); AdamW got a real challenger (Muon / MuonClip on Kimi K2).
- The tail matters. Mid-training on high-quality tokens, long-context extension, pretraining-time distillation for small siblings. The Mamba hybrid is the sole non-attention survivor.
- The model out the other end is still a document completer. Chapter 3 makes it an assistant.
Sources: DeepSeek-V3 (arXiv:2412.19437); DeepSeek-V2 / MLA (arXiv:2405.04434); DeepSeekMoE (arXiv:2401.06066); Mixtral (arXiv:2401.04088); Kimi K2 / MuonClip (arXiv:2507.20534); Muon (arXiv:2502.16982); Llama (arXiv:2302.13971); Llama 3 herd (arXiv:2407.21783); RoPE (arXiv:2104.09864); YaRN (arXiv:2309.00071); MQA (arXiv:1911.02150); GQA (arXiv:2305.13245); FlashAttention (arXiv:2205.14135); FlashAttention-3 (arXiv:2407.08608); DCLM (arXiv:2406.11794); FineWeb / FineWeb-Edu (arXiv:2406.17557); Nemotron-CC (arXiv:2412.02595); phi-1 (arXiv:2306.11644); OLMo 2 (arXiv:2501.00656); Gemma 2 (arXiv:2408.00118); Mamba (arXiv:2312.00752); Jamba (arXiv:2403.19887); Nemotron-H (arXiv:2504.03624); Aux-loss-free balancing (arXiv:2408.15664); FP8 formats (arXiv:2209.05433); Position interpolation (arXiv:2306.15595).