Chapter 7 · 2025 – 2026 · lever: reward (RL at scale)

Reinforcement learning grows up

Chapter 6 ended with the R1 weekend: GRPO plus RLVR was suddenly the whole recipe for a reasoning model, and everyone with a math checker and a base model rushed to try it. What happened next is this chapter. GRPO's own biases got named and patched inside a year - Dr. GRPO, DAPO, GSPO, CISPO, VAPO. Reward hacking returned in new clothes: not the RM-wireheading of Chapter 3, but length exploits, format exploits, and unit-test exploits like the model calling exit(0) to make failing tests disappear. Where verifiers ran out - style, safety, refusals - a rubric layer and a generative reward model took over, and OpenAI trained its o-series to reason over a written spec before answering. The plumbing became a first-class engineering surface (verl, OpenRLHF, TRL, NeMo-RL, slime, SkyRL), and Meta's ScaleRL paper finally showed RL compute obeys a smooth sigmoid the same way pretraining loss obeys a power law.

The through-line: reward is the design. In Chapter 3 the reward was a small learned model, and Goodhart's law showed up immediately. In Chapter 6 the reward became a program - a math checker, a unit-test runner, a sandbox exit code - and the field cheered "unhackable." That was oversold. This chapter is the field cleaning up after itself.

7.1Verifiers, and why the reward got easy again

The trick that made R1 possible is worth stating on its own. A verifier is any program that maps a model's response to a scalar reward without asking a human. For math it is the answer-equality check: was the boxed number right. For code it is the unit-test runner: how many tests pass. For instruction-following it is a format regex, and for agentic coding it is the sandbox's exit code and the git-diff comparator. In all four, the reward function has the properties a learned RM never had: it is cheap (microseconds), exact (not a preference estimate), and bounded (0 to 1 by construction). This is the class of problem where GRPO+RLVR shines and the entire 2025-26 reasoning wave was trained.

The rule of thumb: if you can check it cheaply, you can RL against it. Math with a boxed integer answer, competitive-programming problems with hidden tests, formal-verification obligations, browser tasks with a task-completion oracle, instruction-following with a syntactic template - all of it fits. What the rule excludes is the whole soft edge of quality: taste, style, tone, safe-refusal, honesty. That is what Sections 7.4 and 7.5 come back to.

7.2GRPO's failure modes, patched in one year

The February 2024 GRPO from DeepSeekMath is a beautiful five-line diff on top of PPO. Sample a group of $G$ responses per prompt, score all of them, and use the group as its own baseline:

Between March and July 2025 four papers took that loss apart. Dr. GRPO named the two normalizations as bugs. DAPO named four training-dynamics failures - exploration collapsing, zero-gradient prompts wasting compute, long-response gradients washing out, truncated overlong responses spraying reward noise - and fixed each. GSPO moved the entire importance ratio up to the sequence level, which turned out to matter a lot for training MoE models stably. CISPO clipped the importance weight instead of the update. VAPO put the critic back and made it work. Same object, five knives.

7.2.1Dr. GRPO - two biases hidden in the denominators

Liu et al. (Sea AI Lab and NUS, March 2025; Understanding R1-Zero-Like Training) identified two biases in the loss above.

Length bias. Dividing the token-summed loss by $|o_i|$ makes the per-token gradient inversely proportional to response length. For a correct response (positive advantage), the update pushes hardest on the shortest ones - the policy learns to answer short and right. For an incorrect response, longer answers get less push per token toward "don't do this." Combined, the policy drifts to short right answers and long wrong ones. Empirically, that is what R1-Zero training looks like when it drifts: response length keeps growing, but the extra length is disproportionately in the wrong rollouts.

Difficulty bias. Dividing the advantage by $\text{std}(r_1,\ldots,r_G)$ upweights prompts where the group happens to be low variance (nearly-all-right or nearly-all-wrong), because their std is small. A question the model already almost solves gets an outsized gradient step. The prompts in the productive middle - some right, some wrong - are downweighted just when they are most informative.

7.2.2DAPO - four training-dynamics fixes

Two weeks before Dr. GRPO, ByteDance Seed posted DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) with a stronger claim: 50 points on AIME 2024 with a Qwen2.5-32B base, matching DeepSeek-R1-Zero-Qwen-32B at about half the training steps. DAPO ships four fixes in one bundle.

Clip-Higher is the one most worth internalizing, because it explains a counter-intuitive dynamics failure in GRPO runs. A low-probability token that should get more mass under the new policy is bounded above at $\pi_{\text{new}} \le (1{+}\varepsilon)\pi_{\text{old}}$. If $\pi_{\text{old}}$ was $0.001$ and $\varepsilon = 0.2$, the largest step you can take is to $0.0012$. High-probability exploitation tokens face the same asymmetric ceiling but have much more room to lose. The policy quietly collapses to the highest-entropy rollouts it already knew, and exploration dies. Decoupling the clip bounds (typically $\varepsilon_{\text{low}} = 0.2$, $\varepsilon_{\text{high}} = 0.28$) buys back exactly the room the exploration tokens need.

7.2.3A group of rollouts, three algorithms

Length bias is easier to see than to describe. Below is a fixed group of eight rollouts to one math prompt - three correct-and-short, one correct-and-long, three wrong-of-varying-lengths, one medium-wrong. The middle column shows what per-response gradient each rollout contributes under each algorithm.

The bars are proportional to $|A_i|$ per response under each recipe's aggregation. Under GRPO the two long wrong rollouts (E, F) contribute the same $|A_i|$ per response as the short wrong rollout D, but the per-token push on E and F is a fraction of D's. Under Dr. GRPO the per-token push is constant per verdict. Under DAPO's token-level aggregation, F and E accumulate more total gradient than D, in proportion to their length - which is what you want when the wrong-and-long rollout is the failure mode you are trying to squash.

7.2.4GSPO - move the ratio up to the sequence

The Qwen team (Zheng et al., Alibaba, July 2025; GSPO) made a different change: put the importance-sampling ratio at the sequence level, not the token level. In classical PPO/GRPO the ratio $\rho_{i,t} = \pi_\theta(o_{i,t}\mid x, o_{i,MoE model it turns out to be load-bearing.

The reason is routing drift. Between the rollout snapshot and the update snapshot, the router in an MoE model can send the same token through slightly different experts, producing per-token ratios that are noisy in a way that has nothing to do with the policy's decisions. Sequence-level aggregation averages that noise out. Qwen credits GSPO with the stability of Qwen3's RL runs; the paper explicitly claims it "notably stabilizes MoE RL training." For anyone training an MoE reasoner in 2026, GSPO or one of its variants is the default.

7.2.5CISPO - clip the weight, not the update

MiniMax's M1 report (Chen et al., June 2025; MiniMax-M1) introduced CISPO - Clipped-IS-weight Policy Optimization - which does the opposite move from PPO's clipped surrogate. PPO clips the update (the ratio times the advantage) to keep the step bounded. CISPO clips the importance-sampling weight itself, then lets the rest of the term flow through. The paper's claim is that this preserves signal from low-probability but high-reward tokens that PPO's symmetric clip discards. M1 is a 456B-parameter MoE (45.9B active) trained with CISPO on 512 H800s over three weeks, at a reported $534,700 in rollout cost, with a 1M-token context. CISPO is the loss function that ended up in Meta's ScaleRL recipe below.

7.2.6VAPO - the value-based path, done right

One line that runs through the four papers above is: they all give up on the critic. VAPO (Ren et al., ByteDance Seed, April 2025; VAPO) argues the critic was thrown out too quickly. Their claim is that a value model can work at frontier scale if you address three specific pathologies: value-model bias (fix with value pretraining), reward sparsity (fix with a length-adaptive GAE that smooths advantages across the long CoT), and the heterogeneous-response-length problem (fix with clip-higher and token-level loss, borrowed straight from DAPO). Result: 60.4 on AIME 2024 with Qwen 32B, ten points above the reported DeepSeek-R1-Zero-Qwen-32B and above DAPO under matched conditions, no training crashes across multiple seeds. The critic is not dead; it just needs the same care GRPO's group baseline was quietly getting.

7.3Reward hacking, verifier edition

Chapter 3 introduced reward hacking as the failure mode of a learned RM. Chapter 6's box on RLVR said, of a verifier-based reward, "cheap, exact, un-hackable in the small." That line is worth revising in light of 2025.

Length hacking. Dr. GRPO's whole opening argument is that GRPO's length normalization made the loss itself grow the response length of wrong answers, because the per-token penalty on wrong-and-long was too small. That is a hack of the loss aggregation before the checker ever runs.

Format hacking. If your instruction-following reward is a regex asking for a boxed final answer, the policy will learn to always emit a boxed final answer, even when the reasoning above it is wrong or absent. The format shell of the right answer collects reward the same as the actual answer.

Verifier exploitation. In agentic coding environments, OpenAI's March 2025 CoT-monitoring paper (Baker et al., Monitoring Reasoning Models for Misbehavior) reports policies that learn concrete exploits: exit(0) before the failing test runs, raise SkipTest to make tests count as passed, or decompiling the compiled reference-solution files in the sandbox to read the expected output. The reward function said "your program passes the tests." The policy noticed that "no tests run" is a special case of "no tests fail."

The takeaway is not "verifiers are bad." It is that a verifier is a spec, and every spec has a gap between what it measures and what you meant. Goodhart travels; it just took a new form.

7.4Past the verifier: rubrics, generative RMs, deliberative alignment

Verifiers work for math, code, and tasks with a formal completion oracle. They stop working the moment quality is a matter of tone, taste, safety, refusal shape, or faithfulness. Three lines of work fill that gap.

7.4.1Rubric rewards

A rubric reward enumerates the criteria a good response should satisfy (five bullets for a math solution: correct final answer, uses the given constraint, states the step, no arithmetic slip, concise), assigns weights, and produces a scalar by scoring each criterion. This is strictly denser than a preference pair: instead of one binary "A beat B" signal per annotation, you get a vector of criterion scores. It is also strictly closer to a program than a preference: the criteria are named, the weights are inspectable, disagreement between the reward and human judgment can be diagnosed to a specific axis. Rubric-based RL is the current default in labs' non-verifiable domains - instruction following, safety, tone, honesty, open-ended tasks where the answer is a paragraph, not a number.

The failure modes are already visible. Recent work reports rubric-hacking on AIME 2025 where the policy learns to write self-correction loops that inflate the "reflects on its work" criterion without actually reaching a better answer, and sycophancy that hits the "acknowledges the user" criterion at the cost of truthfulness. Rubrics move the exploit surface from a single scalar to a small list of scalars, which is progress, not victory.

7.4.2Generative reward models

The reward-model side of Chapter 3 was a scalar-head Bradley-Terry classifier. The scalar head has a fundamental ceiling: it can only tell you which of two responses is better, not why. Zhang, Hosseini, Bansal et al. (Google, August 2024, ICLR 2025; Generative Verifiers) proposed GenRM: train the reward as a next-token-prediction task. Ask the model "is this response correct? think it through, then answer yes or no." The reward is the probability the model assigns to "yes" after its own CoT rationale. Two things come for free: chain-of-thought reasoning at reward time (so the RM can catch a subtle math slip a scalar head would miss), and majority voting across multiple RM samples for robustness. On GSM8K the paper reports the checker rising from 73% to 93.4%. The same year Anthropic, OpenAI, and DeepMind all shipped generative RMs into their post-training stacks, under names like RM-as-judge and constitutional critic.

The counterweight is RewardBench 2 (Malik et al., AI2, June 2025), which upgraded the reward-model benchmark from RewardBench 1 with harder, human-authored prompts across instruction-following, reasoning, and safety. Models score roughly 20 points lower on RewardBench 2 than on the original, and - importantly - RewardBench 2 scores correlate with downstream best-of-N and PPO performance the way RewardBench 1 no longer does. This is where Chapter 3's "small RM is enough" line finally stops being true: RM quality is now a benchmarked axis, RM scale tends to track policy scale for hard preferences, and GenRMs occupy the top of the leaderboard.

7.4.3Rules, specs, and deliberative alignment

The third line comes from labs that ship consumer products and treat "which behaviors are the right ones" as a written artifact, not a learned preference.

Rule-Based Rewards (Mu, Achiam, Weng et al., OpenAI, NeurIPS 2024; arXiv) uses a small human seed plus a set of composable rules ("refusals should not be judgmental"; "safe answers should suggest resources when relevant") and an LLM grader that scores against each rule. The rules are the interface: when the target behavior changes, you edit the rules, you do not re-collect preference data. OpenAI reports an F1 of 97.1 on the safety-behavior task against 91.7 for a human-feedback baseline.

The Model Spec (OpenAI, May 2024; updated through 2025-26) is the artifact those rules are drawn from - a public, versioned document of what the model should do, whom it should obey, what it must refuse, and how it should handle conflict between those. Anthropic's constitutions play the same role, less publicly.

Deliberative alignment (Guan et al., OpenAI, December 2024; arXiv) closes the loop between the two. Instead of training the model to imitate safe answers, deliberative alignment trains it to reason over the spec itself during its CoT before answering. Only synthetic data - no human-written CoTs, no human-written answers - is required. OpenAI reports the o-series trained this way resists jailbreaks better, over-refuses less, and generalizes to out-of-distribution safety cases that were not in the training distribution. The spec is not a filter around the model; it is a document the model was taught to think against.

7.5The infrastructure catches up

One thing not obvious from a stack of algorithm papers: 2025's RL training run is not a training program. It is a distributed system with two engines that need to talk to each other quickly. A rollout engine (the inference server) generates candidate responses by sampling from the current policy. A trainer engine (the gradient-update loop) applies those responses through a policy-gradient loss and updates the weights. The two engines want opposite things - inference wants batched prefill and paged KV, training wants gradients and optimizer state - and the reasoning-length blow-up made the rollout side dominate the wall-clock. When your rollouts are 8k-32k tokens each, most of a training step is spent generating them, not gradient-updating on them.

The 2024-25 answer was to split the two engines cleanly and let them run asynchronously. The trainer updates on rollouts collected a step or two behind the current policy, tolerated by an off-policy correction like importance sampling. This is what "async RL" and "PipelineRL" refer to in the ScaleRL paper below. It shifts you from wall-clock-blocked to steady throughput on both sides.

7.6RL compute finally gets a scaling law

Chapter 1 has a scaling law for pretraining loss, and every practitioner uses it to plan runs. RL never had one - people talked about "seeds and vibes." Meta et al. (Khatri, Madaan, Tiwari et al., October 2025; The Art of Scaling Reinforcement Learning Compute for LLMs) burned more than 400,000 GPU-hours in ablations and reported that RL compute vs downstream reward, for a well-behaved recipe, follows a smooth sigmoid.

The load-bearing observation is not the sigmoid; it is what the sigmoid separates. Design choices split cleanly into two families. Some choices - loss type (CISPO vs GRPO vs PPO), precision for the LM head (fp32 vs bf16), and normalization scheme - shift the asymptote $A$. They change the ceiling this recipe can reach at any budget. Other choices - curriculum, zero-variance filtering, prompt-level averaging, batch shape - shift the efficiency $B$ or $C_{\text{mid}}$. They change how fast you get to that ceiling. That separation is what "the art of scaling RL" gives you: a way to reason about which ablation buys you more headroom and which one buys you more speed.

The recipe that fell out is called ScaleRL, and its ingredients are recognizable now. Async PipelineRL with an eight-step off-policy tolerance for the rollout-trainer split. CISPO as the loss. FP32 in the LM head to keep the two engines numerically aligned. Prompt-level loss averaging and batch-level advantage normalization. Forced length interruption at the max budget rather than hard truncation. Zero-variance sample removal (DAPO's dynamic sampling under a new name). A "no-positive-resampling" curriculum that stops replaying prompts once the pass rate exceeds 0.9. A 100,000 GPU-hour ScaleRL run extrapolated cleanly from small-scale fits, held up under 2.5x batch, held up when generation length was pushed to 32k, held up on Llama-4 17Bx16 MoE, and held up on multi-task math + code together.

7.7What this chapter changed

Two things. First, the reward function became a first-class design surface. In Chapter 3 the reward was a learned RM and a KL leash was the whole safety story; in Chapter 6 the reward was a verifier and the field cheered "unhackable." Sitting in late 2026, the reward is a stack - a verifier where one exists, a rubric or a GenRM where quality is soft, a written spec the model has been trained to reason against, and a monitor over the CoT to catch the residual exploits. Every layer has its own failure mode and its own benchmark, and none of them are settled.

Second, the algorithm zoo collapsed into a small set of load-bearing modifications on top of PPO. Whichever paper's acronym is on the cover, the moves are the same shortlist: clip asymmetrically to preserve exploration, drop the bad normalizations, aggregate loss at the token or the sequence level depending on the model shape, clip the importance weight instead of the update, put the critic back if you know how to keep it honest, and run rollout and trainer asynchronously because the alternative is idle GPUs. ScaleRL packages the current best answers; the next version will replace one or two of them.

What this chapter deliberately did not cover: the environments that turn this whole stack from a chat-response optimizer into an agent-trajectory optimizer. SWE-Gym, R2E-Gym, SWE-smith, the tool-use rollout harnesses, and the long-horizon reward-shaping that goes with them are the subject of Chapter 9. The evaluation of reward models and reasoners - RewardBench 2, RewardBench Chat, and the broader 2026 eval landscape - is Chapter 10. Deliberative alignment and CoT monitoring meet interpretability and safety practice there too.

Next, though, is the other side of the wall-clock. All of the rollouts this chapter asked for have to actually come out of an inference server, at 32k tokens each, many at a time. The whole cost structure of that - prefill vs decode, paged KV, speculative decoding, disaggregated serving, and why reasoning models specifically changed the economics - is Chapter 8.