3Teaching it to be helpful

A base language model predicts the next token in a document; it does not know there is a user across the table. The intent gap is the motivation for post-training. This chapter walks the recipe that closed it (RLHF via SFT → reward model → PPO), the product launch that turned it into the iPhone moment of AI, a second alignment line where the labeler is a model reading a written constitution, and one small prompting trick (chain-of-thought) that we will return to twice more in this book.

1.A base model does not know there is a user

Ask a base GPT-3 "Explain photosynthesis to me." A model trained on the open web will often respond by continuing the document. It might produce a list of related homework questions, or a confident non-answer, or nothing at all. It is doing what it was trained to do: predict the next token of plausible internet text. The pretraining objective never mentioned the word user.

Ouyang et al. named the gap directly. The opening line of the InstructGPT paper: "We show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback." The three properties they wanted out the other side (helpful, honest, and harmless, the "3 H's" from Askell et al. 2021) were already in the air, and the paper became the canonical recipe for getting there. (Ouyang et al., 2022; Askell et al., 2021.)

2.The three-stage recipe

InstructGPT's contribution was less an algorithm than a clean sequence. Read the paper for the math; the recipe in three lines is:

Each stage builds on the previous one. Stage 2's comparisons rank the outputs of stage 1's policy, and stage 3's PPO updates stage 1's model using stage 2's reward. Walk through the pipeline below. Click any stage, or hit play.

Stage 1 · SFT, the cheap foundation

Hire about 40 labelers through Upwork and Scale AI. Sample prompts from real API traffic and from the labelers' own imaginations. Have them write the ideal completion, the kind of response a careful, well-informed human would give. Then supervised-fine-tune the base GPT-3 on the resulting (prompt, ideal completion) pairs.

This step alone moves the model a lot. It learns the surface form of helpfulness: start with a direct answer, use the user's terminology, avoid drifting off topic. SFT is by far the cheapest of the three stages and does most of the visible work. The next two stages push past what demonstrations can teach.

Stage 2 · the reward model is a small judge

You can write a few thousand demonstrations. You cannot write a million. To scale, InstructGPT switches from generation to comparison: a labeler is shown 4 to 9 candidate responses to one prompt and ranks them. Comparisons are easier to produce than demonstrations, and they directly capture relative preference, which is what you ultimately want.

A reward model is a transformer with a scalar head. It is trained on the pairwise comparisons with a simple loss: for each pair where y_w was preferred over y_l, push the reward of the winner above the loser. The loss is the Bradley-Terry form, where the reward difference is the log-odds that a labeler will prefer the first response:

$\mathcal{L}_{\text{RM}} = -\log \sigma\!\left(\, \htmlData{tip=scalar reward the RM assigns to the preferred response y_w}{r_\phi(x, y_w)} - \htmlData{tip=scalar reward the RM assigns to the rejected response y_l}{r_\phi(x, y_l)} \,\right)$

One detail: InstructGPT's biggest policy is 175B parameters, but its reward model is only 6B. In 2022 the reward model did not need to write; it only needed to recognize good writing when it saw it. That claim aged unevenly. As preferences got harder (multi-step reasoning, code, long-context critique), frontier practice pushed RM quality up, often close to policy scale, and turned RM performance into a benchmarked axis of its own with RewardBench and its harder 2025 successor. Generative reward models, judges that think out loud in a CoT before producing a score, now beat classifier RMs of the same size on hard problems. Chapter 7 goes into this in detail; for the 2022 recipe, a 6B classifier was enough. (Ouyang 2022, §3.5; Malik et al., RewardBench 2, 2025; Zhang et al., Generative Verifiers, 2024.)

Stage 3 · PPO, with a leash

Now the reward model is frozen and used as the reward function for reinforcement learning. Sample a prompt, have the policy produce a response, score it with the RM, push the policy's weights in the direction that earns more reward. The optimizer is PPO, chosen for its stability under noisy reward signals.

The catch is reward hacking. A policy that purely climbs the RM will drift into word-salad that scores high but reads as gibberish, because the RM has its own blind spots and the policy will find them. So PPO adds a KL penalty to the SFT model: at each step, penalize how far the policy's token distribution has drifted from the SFT policy's distribution.

The result that made the recipe famous

On the labelers' held-out preference ratings:

A 100×-smaller post-trained model was preferred to a giant raw one. Most of the knowledge was already in GPT-3; the missing piece was knowing what was being asked of it. Alignment unlocked existing capability rather than adding new capability, a pattern that will recur. (Ouyang 2022, Fig. 1.)

3.ChatGPT · the iPhone moment

Eight months after the InstructGPT paper landed on arXiv, OpenAI shipped the same recipe on a GPT-3.5 model as a free chat interface. ChatGPT launched on November 30, 2022 as a "research preview." The launch post described the training pipeline as "a method very similar to InstructGPT" with the obvious modification for multi-turn dialogue. (OpenAI, "Introducing ChatGPT," Nov 30 2022.)

What followed was the fastest consumer adoption curve in the history of software. A million users in roughly five days, ~100M monthly actives by January 2023 according to UBS / SimilarWeb estimates reported by Reuters. The model itself was barely different from InstructGPT; the interface and the timing were what changed. (Reuters, Feb 1, 2023.)

4.Replacing the human labeler with the model itself

Every stage of the InstructGPT recipe leans on hired humans. Demonstrations for SFT are written by them. Preference comparisons for the RM are ranked by them. The reward model that PPO then chases is only as good as the ~33k comparisons those forty labelers produced. Scaling this pipeline to a new domain (code, math, long-form writing, safety for a language you don't speak) meant paying more people to rank more outputs. That is the labeler bottleneck.

Anthropic's Constitutional AI (Bai et al., December 2022) proposed a way around it: hand the ranking job to the model itself, steered by a short written constitution of principles. The recipe has two stages. First, "SL-CAI": the assistant produces a response, the model critiques its own response against the constitution, revises it, and is then supervised-fine-tuned on those revisions. Second, "RL from AI Feedback" (RLAIF): for each prompt, sample two responses, ask an AI evaluator (prompted with the same principles) which one is better, train a preference model on those AI-generated labels, and run PPO against it. The claim in the abstract is the striking one: "we are able to train a harmless but non-evasive AI assistant [...] without any human labels identifying harmful outputs." (Bai et al., 2022.)

Google made the head-to-head comparison a year later. Lee et al. found that on summarization and helpful-dialogue, RLAIF was comparable to RLHF even when the AI labeler was the same size as the policy, and they showed a stripped-down variant called direct-RLAIF, where the LLM produces the reward online during RL with no separately-trained RM at all. Human preference labels stopped being the only source of alignment signal. (Lee et al., 2023.)

5.Chain-of-thought: reasoning falls out of a prompt change

Five weeks before InstructGPT, a different paper out of Google Brain (Wei, Wang, Schuurmans, and colleagues) reported a simple result. If you give a large language model a few worked examples of step-by-step reasoning and then ask it your question, its accuracy on multi-step problems goes up sharply, with no weights touched. (Wei et al., 2022.)

The change is small. Prepend a few demonstrations like "Q: ... A: First we ... then we ... so the answer is 27." instead of "Q: ... A: 27." The model picks up the pattern and produces its own intermediate steps before committing to a final answer. This is chain-of-thought prompting. The improvement is largest on the tasks where reasoning steps actually help: arithmetic, multi-step word problems, symbolic manipulation.

The headline number from the paper: PaLM 540B with 8-shot chain-of-thought reached 56.9% on GSM8K, where the prior state-of-the-art (a fine-tuned GPT-3 with a learned verifier) was 55%. With an external calculator the same setup pushed to 58.6%. A prompt change matched and then beat a system that had touched weights.

A second observation in the paper matters more than the headline: chain-of-thought only helps once the model is large enough. Below roughly 100B parameters, asking for reasoning steps mostly hurts. Above it, the gap opens up. Wei and colleagues named the pattern a few months later in a companion paper: emergent ability, a capability that appears at scale rather than improving smoothly. (Wei et al., "Emergent Abilities of Large Language Models," 2022.)

BBH: the eval bed for the seed

Numbers on GSM8K alone are not enough to show that reasoning has been unlocked broadly. Later in 2022, Suzgun et al. cut a 23-task subset out of BIG-Bench called BIG-Bench Hard (BBH), chosen because prior model evaluations had failed to beat the average human rater on them. With plain prompting, large models still trailed humans. Add chain-of-thought, and Codex (code-davinci-002) surpassed the average human on 17 of 23 tasks; PaLM 540B on 10 of 23. (Suzgun et al., 2022.)

BBH will show up as the eval shadow over the next three chapters. "+X points on BBH" appears in almost every prompt-optimization paper from 2022 onward. The benchmark's job is to be a hard-enough bar that prompt-level interventions can register as real progress.

6.The CoT seed grows twice more in this book

Chain-of-thought arrives here as a prompt trick with no weights touched. That framing grows twice. In Chapter 5 the same idea becomes an optimization target: GEPA and its cousins search for the best CoT-shaped prompt automatically, treating wording as a parameter you can optimize. In Chapter 6 the idea moves directly into the weights: o1 and DeepSeek-R1 train models to produce long internal reasoning traces by reinforcement learning on verifiable rewards. "Make the model show its work" surfaces at three different layers of the stack, and this is the first.

7.The recipe works, and it is expensive to run

Read the InstructGPT paper carefully and the cost shows up everywhere. Forty labelers on contract. ~13,000 demonstrations. ~33,000 ranked comparisons. Three sequential training runs, each on top of the previous one. PPO is high-variance in practice, and small hyperparameter changes wreck a run. You need a frozen reference policy in memory to compute the KL penalty, a separate critic to estimate value, and the reward model alongside the policy you are actually updating: four models loaded at once, plus an RL loop on top.

The next chapter follows the cost curve as it breaks downward. DPO drops the reward model and the RL loop, learning directly from preference pairs with a single classification loss; GRPO keeps RL but drops the critic by using a sampled group as its own baseline. The alignment target is the same; the machinery underneath gets much smaller.