Chapter 0 ended with a bet: one architecture plus pretraining could replace the zoo of bespoke models. This chapter is what happens when you press the "more" button on that bet for three years. The deliverable is not a smarter model - it is a smarter way to plan. By the end, prompting has replaced fine-tuning as the default interface, capability has a power-law graph, Chinchilla has caught the field undertraining its giants, and the transformer has crossed into vision. The chapter closes by breaking its own rule: Chinchilla was the answer for training compute and turned out to be the wrong answer once you had to deploy the model, which is what Chapter 2 picks up.
1. The shift in posture
The headline finding of Brown et al. 2020 is buried in one line of their abstract: GPT-3 "is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." Read that twice. The model is frozen. The task is described in English. The training signal - if you can still call it that - is a handful of examples typed into the context window. This is in-context learning, and it broke the loop everyone had been running since the BERT era.
Before GPT-3, the recipe for a new task was: take a pretrained model, fine-tune it on your labels, ship the per-task checkpoint. After GPT-3, the recipe was: take the one model, describe the task in English, paste a few examples in front of your input, read the output. The verb changed. You no longer train a model to do a task. You prompt one.
Programming a model became writing for a model. That is the sentence the rest of this course is a slow-motion reaction to. Once the prompt is the interface, "the prompt" becomes something worth studying, something worth optimizing, eventually something worth searching over. Chapters 3, 5, and 11 all live in this consequence.
1.1 In-context learning, on one screen
Same instruction, same query, three different prompts - the only thing that changes is how many demonstrations sit in the context. The model output is canned for this widget; the point is the shape of the conditioning, not a live inference.
Few-shot prompting is not "the model memorized this in pretraining." Brown et al. showed the effect grows monotonically with the number of demonstrations and with model scale, on tasks engineered to be novel. The base model already contains the competence; the demonstrations choose which competence to evoke. Think of the prompt as a selector over a huge superposition of latent capabilities.
2. The shape of the bet
The shock of GPT-3 was not just what it could do - it was that the result was predictable. Six months earlier, Kaplan et al. 2020 had published the curves. Loss as a power law in compute. Loss as a power law in parameters. Loss as a power law in data. Each clean, each spanning multiple orders of magnitude. From the abstract, verbatim: "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."
A relationship of the form $L(C) = A \cdot C^{-\alpha}$ - a straight line on log-log axes. The exponent $\alpha$ tells you how much loss drops per decade of compute. Kaplan reports $\alpha_C \approx 0.05$ for compute, $\alpha_N \approx 0.076$ for parameters, $\alpha_D \approx 0.095$ for data, fit across more than seven orders of magnitude.
What is load-bearing here is not the specific exponent. It is the existence of the line. Before scaling laws, "make the model smarter" was research alchemy: try things, see what works, publish. After scaling laws, it is a budget calculation. Pick a compute budget, read the corresponding loss off the line, decide if the gain is worth the money. Capability becomes a function you can plan against.
GPT-3 was the existence proof attached to that line. 175 billion parameters - "10x more than any previous non-sparse language model," in the paper's own framing - trained on hundreds of billions of tokens, sampled with weighting from a corpus of roughly 570B tokens. The compute bill was on the order of thousands of petaflop/s-days. Numbers that would have sounded absurd in 2019 read, in retrospect, as exactly the dose Kaplan's exponents predicted.
2.1 The line, on log-log axes
Slide the budget. Both curves obey the same power law - $L = A \cdot C^{-\alpha}$ - with the same exponent. They differ only in their prefactor $A$, which is set by how you spend the budget across parameters and tokens.
The orange curve traces Kaplan's prescription: pour most of the budget into parameters, keep data modest, stop before convergence. The teal curve traces what Chinchilla would later find is actually compute-optimal: scale params and tokens together. Same exponent, lower prefactor, lower loss at every budget. The gap is the chapter's pivot - and the next section's subject.
3. Chinchilla, or: eat your spinach
Two years of giants followed Kaplan. Gopher (280B). Megatron-Turing NLG (530B). Jurassic-1 (178B). All of them training on token counts roughly comparable to GPT-3's. All of them, it turned out, undertrained.
Hoffmann et al. 2022 at DeepMind trained more than 400 models across a wide range of (params, tokens) combinations, fit a joint scaling law to all of them, and read off the compute-optimal frontier. Their punchline: for every doubling of model size, the number of training tokens should also double. Params and data scale together. Kaplan's "spend on params, hold data modest" was an artifact of the corner of the budget surface he had sampled.
The proof was a model named Chinchilla: 70B params, 1.4T tokens - same compute budget as Gopher's 280B params on 300B tokens. Quarter the parameters, four times the data. Same money. The 70B model uniformly and significantly outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing (530B), including 67.5% on MMLU - seven points above Gopher.
Not the existence of the scaling law. Just the ratio. The 2020 giants had been overweighting params and underweighting data. Re-anchored on the right ratio, the same hardware buys a noticeably better model. After Chinchilla, every serious lab refits its data pipeline first and its parameter count second. LLaMA 1, Mistral, and the whole open-weights wave that arrives in Chapter 4 all sit downstream of the 70B / 1.4T ratio.
3.1 The two prescriptions, side by side
| Rule | Kaplan (2020) | Chinchilla (2022) |
|---|---|---|
| Where to spend compute | Mostly parameters | Parameters and tokens equally |
| Train to convergence? | No - stop early | Yes - feed more tokens |
| Reference 70B model would use | ~140B tokens | ~1.4T tokens |
| Verdict, in hindsight | Undertrained | Compute-optimal |
A subtle but important thing the table does not show: Chinchilla did not refute Kaplan, it refit him. The power-law form survives. The mistake was in the corner of (N, D) space that had been sampled. The lesson is methodological as much as numerical - if your scaling law was fit on undertrained models, do not extrapolate it.
3.2 Compute-optimal is not deployment-optimal
A frontier lab in 2026 does not train Chinchilla-optimal models, and it is worth being explicit about why - because "Chinchilla is the answer" is where the reader would be if the chapter ended at the table above. Chinchilla answers a narrow question: for a fixed pool of training compute, what (N, D) split minimizes loss? A production model spends most of its life doing inference. Once you write down the joint cost of training plus a realistic amount of serving, the optimal shifts.
Sardana et al. (2023) did that arithmetic. Their finding: if you expect meaningful inference demand, train models smaller and longer than Chinchilla says. A smaller model fed extra tokens catches the same loss as a bigger Chinchilla-optimal model, at higher training cost, but cheaper every time it answers a query afterward. Their fits push token-per-parameter ratios into the four-digit range for inference-dominant regimes; Chinchilla's own ratio is roughly 20.
The empirical answer showed up in Llama 3 (Grattafiori et al., 2024). The 8B model is trained on 15 trillion tokens - about 1,875 tokens per parameter, roughly two orders of magnitude past Chinchilla-optimal - and Meta explicitly notes that loss was still improving log-linearly when they stopped. Chinchilla says you should not do that. Inference math says you absolutely should.
Chinchilla is compute-optimal for training; frontier labs now over-train smaller models because deployment cost dwarfs training cost. The scaling law still holds. The objective people optimize is different.
3.3 The data wall
Both Chinchilla and its inference-aware successor assume tokens are available for the taking. That assumption is starting to buckle. Muennighoff et al. (2023) rewrote the scaling law for the case where the dataset is finite, showing that each additional epoch of repeated data is worth less than the last and pinning numbers on the decay: roughly four epochs of repeats still help; twenty do not. The corrected law re-prices every extrapolation that assumed you could just keep streaming fresh tokens.
Villalobos et al. (2024) then tried to measure how many tokens actually exist. Their estimate: the effective stock of quality-adjusted public human text is on the order of 300 trillion tokens, and frontier training runs will exhaust it somewhere between 2026 and 2032 - earlier if over-training keeps intensifying. That is the data wall. It is why every frontier lab now spends serious effort on filtering (DCLM, FineWeb-Edu, Nemotron-CC), on synthetic data pipelines (the phi line), and on reasoning-model self-play - all of which Chapter 2 takes apart.
4. Naming the noun: foundation models
By mid-2021, a strange thing had happened. The same pretrained model was being adapted - via fine-tuning, prompting, or both - for translation, summarization, code generation, search, classification, and a long tail of bespoke tasks. One model underneath, hundreds of applications on top. The field needed a noun for that thing.
The Stanford CRFM report (Bommasani et al. 2021, over one hundred authors) crystallized it. From the opening line: "AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character." The label has stuck.
"Foundation model" was popularized, not coined, by the CRFM report - the term had appeared in adjacent contexts before. Safer phrasing for the historical record: the noun was crystallized by Stanford in August 2021. What mattered for the field was less the etymology and more that everyone now had the same word for the substrate.
5. The transformer walks into vision
The same two years quietly settled an older debate. Convolutions had ruled vision since 2012; attention had ruled NLP since 2017. Two papers stapled them together by noticing that the staple was unnecessary.
ViT (Dosovitskiy et al. 2020): split an image into 16x16 pixel patches, treat each patch as a token, feed them into a vanilla transformer encoder. From the paper: "a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks." With enough pretraining data, ViT matched or beat the best CNNs - using none of the convolutional inductive bias people had spent a decade engineering in.
CLIP (Radford et al. 2021) went further. Scrape 400 million (image, caption) pairs from the web. Train one encoder for images and one for text using a contrastive objective - pull matching pairs together in a shared embedding space, push mismatched ones apart. The result was zero-shot ImageNet at ResNet-50 accuracy, without ever seeing the 1.28M ImageNet training images. Vision had learned to speak the same prompt-able dialect.
If your last touchpoint with vision was 2019 ResNets and EfficientNets, this is the era where the convnet stops being the only game. ViT does not retire the CNN - convolutions remain excellent on small-data regimes - but it shows the same architecture from Chapter 0 generalizes across modalities once you give it enough data. Multimodality is a recurring thread, not a separate chapter. It returns in Chapter 11 when the native-multimodal frontier shows up.
6. The gap that opens Chapter 2
The chapter has done its scale story twice over: capability follows a power law, and the ratio of parameters to tokens is the knob that decides how efficient the spend is. But the last two subsections just showed that the "compute-optimal" answer is compute-optimal for a fake objective. A real production model has to live somewhere, be served to millions of requests, run inside a memory and latency budget. And the token stream that feeds all of it is finite.
That reframes the question the field spent 2022 answering. It is no longer how do I hit a target loss with the least training compute. It is how do I hit a target quality with the least lifetime cost, on data I have to curate hard because there is not much of it left. The Chinchilla ratio is a rung on that ladder, not the top.
Chapter 2 is the answer to that reframed question. It walks through what a frontier pretraining run actually spends its budget on in 2026 - the data pipeline (filtering, dedup, decontamination, synthetic augmentation), the architecture updates the 2017 block has absorbed (MoE, MLA/GQA, RoPE + long-context stages, RMSNorm, SwiGLU), the numerics (BF16 master weights and FP8 forward passes), the optimizer beyond AdamW (Muon and MuonClip), and the parallelism story that makes any of it fit. A base model that comes out the other side of that pipeline is still a document completer. Making it helpful is Chapter 3.
Chapter 0 turned the architecture lever - one stack of self-attention replaced a zoo. Chapter 1 turned the scale lever and then discovered the objective it was optimizing was too narrow. Chapter 2 turns the efficiency lever - the same compute, spent better on data, architecture, and numerics. Same story of levers, different rung.
7. What to take with you
- Prompting replaced fine-tuning as the default interface. Few-shot examples in the context window steer a frozen model. The verb changed from "train" to "prompt."
- Capability became predictable. Loss is a power law in compute, params, and data. Bigger-is-better stops being a hunch; it is a graph you can read.
- The ratio matters, but not the one Chinchilla published. The 2020-era giants were undertrained; Chinchilla fixed the training-compute ratio; inference cost then re-fixed it in the other direction, and the data wall is starting to fix it a third time.
- One model became the substrate. "Foundation model" is the noun the field needed once everything downstream was a thin wrapper on the same pretrained backbone.
- Vision joined. ViT and CLIP show the transformer is not a text architecture - it is a token architecture. Patches are tokens.
- The next lever is efficiency. Chapter 2 opens the hood on a modern pretraining run: data pipeline, MoE, MLA/GQA, FP8, Muon, the whole stack that fits under "spend the compute better."
Sources: Brown et al. 2020 (arXiv:2005.14165); Kaplan et al. 2020 (arXiv:2001.08361); Hoffmann et al. 2022 (arXiv:2203.15556); Sardana et al. 2023 (arXiv:2401.00448); Muennighoff et al. 2023 (arXiv:2305.16264); Villalobos et al. 2024 (arXiv:2211.04325); Grattafiori et al. 2024 (arXiv:2407.21783); Bommasani et al. 2021 (arXiv:2108.07258); Dosovitskiy et al. 2020 (arXiv:2010.11929); Radford et al. 2021 (arXiv:2103.00020).