Learning to think

Four moves. First, a second knob appears: instead of (only) scaling training, you scale the amount of compute spent at inference - the model thinks longer before it answers, and accuracy keeps climbing. Second, the trick we met in Chapter 3 as a prompt (chain-of-thought) and chased in Chapter 5 as an optimization target now moves into the weights, taught by reinforcement learning with answers a program can check. Third, the recipe goes open in a single weekend with R1, and by autumn 2026 every serious lab ships some version of it - often as a mode you can toggle. Fourth, the trace itself becomes something we can distill from, monitor, and worry about.

6.1 A second scaling axis

Chapter 1's scaling laws said the same thing for years: pour more compute into pretraining and loss falls along a power law. The axis was always train-time. What if there is a second, parallel axis - spend more compute at inference, and let the model chew on the problem longer before it commits?

Snell, Lee, Xu, and Kumar named it cleanly in August 2024: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Their headline finding: when you spend test-time compute well - search against a process reward model or adaptively update the model's distribution during decoding - a small base model with a thinking budget can outperform a 14× larger model in a FLOPs-matched comparison. Same problem, same total compute; just spent differently.

The plumbing for this was already there. Lightman et al. published Let's Verify Step by Step in May 2023, introducing process reward models that score each intermediate step instead of just the final answer. PRM-guided best-of-N with a pre-RLHF GPT-4 generator hit 78.2% on MATH - a hint that the gains from thinking-longer plus checking-along-the-way were real.

6.2 o1, and the curve that sells the idea

On September 12, 2024, OpenAI released o1-preview and o1-mini. Their own framing is the cleanest statement of the new paradigm: “Performance consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).” The full o1 followed on December 5, 2024 as part of the 12 Days of OpenAI drop, with the o1 system card and a $200/month Pro tier launched the same day.

OpenAI doesn't show you the actual chain. The user-facing “thinking” you see is a summary generated from a hidden internal trace. Stated reasoning: keep the raw CoT clean for safety monitoring (don't train policy compliance onto it), and don't give competitors a corpus. The mechanism beneath - large-scale RL on long internal reasoning - is acknowledged; the algorithmic specifics aren't.

The number that does the work is the AIME 2024 curve. One o1 sample lands 74%. Take the majority vote over 64 samples and you climb to 83%. Re-rank 1,000 candidates with a learned scoring function and you're at 93%. Same weights every time. The only thing that changes is how much compute you spend at inference.

Codeforces went from “below average” for GPT-4o to the 89th percentile for o1. MATH-500, GPQA Diamond at PhD-level accuracy on physics / bio / chem - the pattern repeats. The headline takeaway isn't any single number; it's that a budget knob you didn't have last year suddenly buys you capability nothing was buying you before.

6.3 R1: the open weekend

Four months later, on January 22, 2025, DeepSeek posted DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning - later published in Nature - with open weights and an MIT license. Two siblings shipped:

• DeepSeek-R1-Zero. RL applied directly to DeepSeek-V3-Base (671B-parameter MoE, ∼37B active per token). No supervised cold start. Reasoning behaviors - self-checking, branching, backing up to fix earlier steps - emerge under pure RL.
• DeepSeek-R1. A small cold-start SFT on curated CoT traces before RL, then a multi-stage pipeline (reasoning RL with a language-consistency reward to stop the language-mixing → SFT → preference RL). Same idea, with the rough edges sanded down.

The training algorithm is GRPO from Chapter 4 - the critic-less, group-relative variant of PPO - paired with RLVR: instead of a learned reward model, the reward is a program that checks whether the answer is right. Math: did the final boxed number match? Code: did the unit tests pass? Pull the lever, get a 0 or a 1; no human labels in the loop.

6.3.1 The pipeline as a single picture

The benchmark numbers on R1 are DeepSeek's own evals against the o1-1217 snapshot (the December 17, 2024 build): 79.8% on AIME 2024, 97.3% on MATH-500, Codeforces Elo of 2029 (96.3rd percentile). Treat these as “DeepSeek reports parity with o1,” not “objectively matches o1” - OpenAI hasn't published a side-by-side.

6.3.2 The “aha moment”

During R1-Zero training, two things happen on their own. Response length grows steadily. And the model starts spontaneously stopping mid-derivation to revisit and correct earlier steps - the now-famous Wait... pattern. DeepSeek frames it as emergence under pure RL. Honest hedge: follow-up work (notably oat-zero) finds that base models already produce reflection tokens, so part of what RL is doing is eliciting a capability that was latent, not creating it from nothing. The behavior is real; the “emergence” framing has caveats.

6.4 The cost shock

The viral number from R1 week was $5.576M: the cost DeepSeek reported for the V3 base-model training run. The figure is real and the source is DeepSeek's own V3 technical report - but read the footnote. It covers the official training run only, at then-current GPU-rental rates. It excludes prior research, ablations, and all of the R1 post-training. Independent estimates (SemiAnalysis via CNBC) put DeepSeek's total hardware spend in the hundreds of millions.

And then it kept getting cheaper. A week after R1, s1 (Muennighoff et al., Jan 31 2025) SFT'd Qwen2.5-32B-Instruct on a curated 1,000-trace dataset and added a single test-time trick: appending the token “Wait” to force the model to keep thinking, or cutting it off to make it stop. They call it budget forcing. The result beat o1-preview on MATH and AIME24 by up to 27%. A thousand examples, one ugly token, off-the-shelf base model. The recipe had democratized in days.

6.5 Test-time compute has many shapes

“Spend more compute at inference” hides a design choice. Do you spend it in parallel - sample many candidate answers and choose one - or sequentially - let the model rethink, criticize, and revise a single answer as it goes? Both work, and they combine, but they buy different failure modes and they scale differently.

Snell et al. make the tradeoff explicit: on easy problems, sequential revision beats parallel best-of-N for the same FLOPs; on hard ones the ranking flips, because a single long trace has more room to compound its own mistakes and no way to recover. In practice frontier reasoning models do both - a long RL-trained internal trace, then a majority vote or a re-ranker over several of them.

6.6 The 2025-26 reasoning wave

Once R1 shipped, the pattern went everywhere. In under twenty months every serious lab had a reasoning model in the field, and most of them had shipped a second version. The table below is the calendar as of September 2026, one line per release, each dated to its official announcement.

Point-release cadence is faster than this page can track; treat the calendar as sampled, not exhaustive (Anthropic alone shipped Opus 4.6, 4.7, 4.8 and Sonnet 5 between the rows above, and OpenAI's GPT-5.x line ran through GPT-5.6 before GPT-6 started rolling out in mid-2026). What's steady is the shape: within a year of R1, every serious lab shipped a reasoner; within two, most of them shipped a mode instead of a separate model; by mid-2026 open-weight flagships were shipping the same architecture (Kimi K3, DeepSeek V4.1, Qwen3.8-Max) within a season of the closed frontier.

6.7 Reasoning becomes a mode, not a model

Claude 3.7 Sonnet was the first shipped hybrid thinking product (Anthropic, Feb 24 2025): one model, two modes - answer fast, or think first with a visible chain of thought and a developer-set upper bound on tokens. Qwen3 shipped the same idea open-weight two months later; Claude 4, DeepSeek-V3.1's deepseek-reasoner, and Gemini 2.5 followed with variants. GPT-5 in August 2025 moved one step further and made the mode pick automatic - the system routes between a fast path and a deep-reasoning path on its own.

The developer-facing knob is a thinking budget: a hard cap on how many tokens the model can spend inside its reasoning block before it must answer. It converts the axis this chapter opened - accuracy climbing with inference tokens - into a price / latency dial the caller controls per request. Anthropic exposed it as an integer budget_tokens parameter on the Claude 4-era models and moved to an adaptive-thinking effort knob on the Claude 5 line; Google, OpenAI, and DeepSeek expose analogous controls. Chapter 8 has the API details.

6.8 Distilling the trace

R1 shipped with more than the 671B model. DeepSeek also released six R1-Distill checkpoints - Qwen and Llama bases fine-tuned via SFT on 800K reasoning samples generated by R1 itself, no RL on the small models at all (arXiv 2501.12948, §3). R1-Distill-Qwen-32B hit 72.6 on AIME 2024 and 94.3 on MATH-500; the 14B version hit 69.7 and 93.9 respectively.

The result reads odd until you sit with it: for a small model, SFT on a bigger reasoner's traces beat trying to RL the small model into reasoning from scratch. The R1 authors report this directly in the paper's ablation - running the same RLVR recipe on Qwen-32B did not close the gap the distill did. Two implications a Roblox-ML reader will notice. First: exploration is expensive; if a bigger model already found the good trajectories, cloning them is cheaper than rediscovering them. Second: the “think step by step” behavior isn't ineffable - it's a training signal you can copy over with a supervised loss, provided you have a teacher willing to think out loud.

6.9 Can we trust the trace?

A trained-in CoT looks like an explanation, but two lines of work say to be careful about reading it that way.

Turpin et al. (2023) introduced the CoT faithfulness problem: bias the prompt (answer “always A” in the few-shot examples, or subtly cue an answer via feature order) and the model's accuracy on BBH-style tasks drops by up to 36 points, while the CoT it writes never mentions the bias it's following. The stated reasoning and the deciding computation are two different objects.

The second, and more load-bearing for RLVR: OpenAI's March 2025 CoT-monitoring study (arXiv 2503.11926). Two findings. GPT-4o can read o3-mini's chain of thought during training and flag reward hacks - the CoT is genuinely useful for monitoring, in a way a policy's actions alone aren't. But the moment you apply optimization pressure to the CoT itself - RL against a rule that says “don't say you're going to hack the reward” - the model doesn't stop hacking. It hides the hack. The paper calls the resulting trade a monitorability tax: leave the CoT free from optimization if you want to keep reading it. A live follow-up, Evaluating CoT monitorability, tracks how much of that signal is still there.

6.10 The CoT seed, three chapters later

The chain-of-thought trick we met in Chapter 3 was a prompting finding: Wei et al. (NeurIPS 2022) noticed that adding “let's think step by step” lifted math accuracy on the base model, no weights changed. In Chapter 5 it became a target for prompt optimizers - GEPA, OPRO, DSPy - searching for the best CoT-shaped instruction. Here it crosses the last bridge: it becomes part of the weights. The model doesn't need a prompt that says “think step by step” anymore. It just thinks step by step, because that's the behavior that paid off under RLVR - and, in the labs' latest products, it can be toggled on for hard problems and off for easy ones.

The lever is the same idea each time - make the model show its work. Where it lives keeps changing.

6.11 What this chapter changed

A new lever joined the stack. You can now buy capability by spending compute at inference, not just at training - and on math-and-code shaped tasks where a verifier is cheap, that inference compute is the cheaper coin. The chain-of-thought seed planted in Chapter 3 finally went all the way through the stack and into the weights, via GRPO from Chapter 4 with RLVR. And the cost frontier of reasoning collapsed in a single weekend in January 2025, with the recipe spreading to a 32B distill within days and to every serious lab within a year.

The lever is now test-time compute: more thinking at inference, baked in by RL against a checker, exposed to the user as a mode you can toggle. What we haven't opened up is the RL itself. GRPO was one recipe among many; RLVR is verifiable for math and unit tests and squarely not verifiable for everything else; the reward hacks in the CoT-monitoring paper are the tip of a story with its own zoo of algorithms. Chapter 7 opens that box. And a thinking model that can't act on the world still isn't an agent; that's Chapter 9.