8Paying for thought

The RL work in Chapter 7 pushed the cost of frontier capability off the training bill and onto the inference bill. A single reasoning-model call can now emit tens of thousands of tokens before it answers, and every one of them has to be produced by hardware someone pays for. This chapter is the systems layer that made that math work. We start with the two very different phases of a transformer forward pass (prefill and decode) and the KV cache that connects them. Then the engineering that grew up around each: PagedAttention and continuous batching for the cache, speculative decoding and quantization for the arithmetic, prefix caching for the conversation, and disaggregated prefill/decode plus expert parallelism for the largest deployments. We close on what long reasoning traces did to the economics, and hand off to agents in Chapter 9, where the same tokens get spent inside a loop.

8.1 Two phases of a token

A generation request splits into two very different pieces of work. Given a prompt of length P and a target output of length D, the model first runs prefill: a single forward pass over all P input tokens in parallel, which fills the attention KV cache for every layer and produces the first output token. Then decode takes over: D−1 more forward passes, each producing exactly one token, each reading the entire KV cache to attend over everything so far.

These two phases have opposite hardware profiles. Prefill has P tokens of work per layer; a matmul against the weights is compute-bound, and an H100 or H200 can chew through thousands of tokens per second per replica. Decode has one token of work per step; the matmul is skinny, GPU compute sits idle, and the bottleneck is HBM bandwidth. Each step must stream the full weights and the full KV cache from memory to registers just to produce one token. The Splitwise paper (Patel et al., Microsoft/UW, Nov 2023) characterized it cleanly: prefill saturates FLOPs, decode saturates bandwidth, and the two happening on the same GPU fight each other for both.

8.2 The KV cache, in bytes per request

The KV cache is a per-request buffer that grows by one entry per token per layer. Chapter 2 walks through the per-variant arithmetic: MHA stores 2 × n_heads × head_dim elements per token per layer, GQA (Ainslie et al., 2023) shares KV across query heads and drops the n_heads to n_kv_heads, MLA (DeepSeek-V2) caches a compressed latent and reconstructs K and V on the fly. Take that per-token byte count as given. This chapter's problem is what the serving layer does with it.

A single Llama 3 70B request at 128K context (GQA-8, BF16 KV) needs about 40 GB of cache. DeepSeek-V3 at the same context, in BF16 KV, needs about 9 GB: the entire difference is MLA plus V3's smaller n_layers. But one request is not what a node serves. The interesting number is how many concurrent requests fit alongside the model weights on the HBM you paid for. That is where the memory manager (PagedAttention), the KV dtype, and the cache-sharing story make or break the deployment:

The three rows compare the same node under three serving disciplines: reserve-the-max static allocation (pre-vLLM defaults), PagedAttention (only pay for the tokens actually used), and PagedAttention plus FP8/INT8 KV-cache quantization. On the 70B node at 128K, moving from naive to paged roughly doubles concurrency; adding FP8 KV doubles it again. The dollar cost per served token falls by the same factor.

8.3 PagedAttention and continuous batching

The first serving-side breakthrough was PagedAttention, from Kwon et al. (SOSP 2023), released as the vLLM engine. The observation is embarrassingly simple once stated: pre-2023 serving systems allocated each request a contiguous KV buffer sized for the model's maximum context length. Real requests are shorter and varied, so 60–80% of that reserved memory sat empty. Sequences that could have fit did not, because the memory manager could not compact around them.

PagedAttention borrows the operating-system trick: cut the KV cache into fixed-size blocks (16 tokens is typical) and keep a per-request block table that maps logical positions to physical blocks. Blocks can live anywhere; new blocks are allocated on demand. The attention kernel is modified to gather from a scattered layout. Fragmentation drops to a few percent, and the effective batch size the GPU can hold climbs sharply. The paper reports 2-4× throughput over FasterTransformer and Orca at matched latency on LLaMA-family models; a widely cited Anyscale benchmark from June 2023 put the improvement over naive Hugging Face static batching at closer to 23×.

The second lever, applied in the same engine, is continuous batching (the term Anyscale popularized in mid-2023 for the iteration-level scheduling Orca (Yu et al., OSDI 2022) first shipped). Static batching waits for the longest sequence in the batch to finish before serving the next request; continuous batching lets finished sequences leave the batch mid-step, and new requests join at the next decode iteration. Combined with paged memory, the GPU no longer idles while one long sequence finishes, which is where most of the throughput gain comes from.

8.4 Prefix caching

Chat traffic has a hidden structural gift: most requests share a long prefix. A system prompt, a tool schema, a few-shot template, or an in-context document: the leading thousands of tokens are the same across many calls. Prefill for those tokens is pure waste if you have already done it once.

Prefix caching keeps the KV cache produced by past requests around and reuses it whenever a new request starts with the same tokens. vLLM ships this as automatic prefix caching: a hash of the block's content is the lookup key, and cache hits skip prefill entirely for the shared prefix. SGLang generalized it further with RadixAttention (Zheng et al., SGLang, Dec 2023), which organizes the cache as a radix tree so branching prompts share their common trunk instead of only linear prefixes. For agent workloads that spawn many sibling calls off a shared plan, radix-style sharing changes the cost curve outright.

Prefix caching is one of the reasons API prices for cached input tokens are typically ~10% of the uncached rate. The provider spent zero flops recomputing that prefix, and the savings are (partially) passed on.

8.5 Speculative decoding

Decode is bandwidth-bound, which means a single decode step under-uses the GPU's arithmetic. That slack is where speculative decoding (Leviathan et al., 2022) and Chen et al. (2023, DeepMind) live. Use a small, cheap draft model to propose the next k tokens, then verify all k in a single parallel forward pass of the big target model. Every drafted token that matches what the target would have produced is a token accepted for free. The output distribution is provably identical to running the target alone (there is a rejection-sampling rule that preserves it, even when the draft disagrees). No quality loss; only fewer target forward passes per token.

The whole trick lives inside two constants: acceptance rate $\alpha$ (fraction of drafted tokens the target keeps) and cost ratio $c$ (draft-forward time over target-forward time). Leviathan's Theorem 3.8 puts the expected speedup over plain decoding at $\htmlData{tip=numerator: expected tokens accepted per verify pass; denominator: cost of one target step plus k draft steps}{\frac{1 - \alpha^{k+1}}{(1 - \alpha)(1 + c k)}}$. The two failure modes: a draft too slow (large $c$) eats the win; a draft too weak (small $\alpha$) rarely gets its predictions accepted. So the practical research has been about better drafts.

• EAGLE / EAGLE-2 / EAGLE-3 (Li et al., Jan 2024; June 2024; March 2025). A tiny autoregressive head reused on top of the target's own penultimate-layer hidden features drafts far better than a separate model, because it sees almost exactly what the target sees. EAGLE-2 adds dynamic tree drafting; EAGLE-3 removes the constraint that the draft head predict features and lets it predict tokens directly. Reported speedups climb from ~3× on Llama-family targets in EAGLE-1 to ~4× in EAGLE-2 and up to ~6× in EAGLE-3.
• Medusa (Cai et al., Jan 2024). Multiple parallel decoding heads on top of the target predict tokens at offsets +1, +2, +3, +4 in one shot; a tree-attention pass verifies the joint hypothesis. Simpler than EAGLE; used as a baseline.
• MTP-as-drafter. DeepSeek-V3 trains multi-token-prediction heads as an auxiliary objective during pretraining (following Gloeckle et al., Meta, 2024). At serving time those heads double as an EAGLE-style drafter, costing nothing extra to train.

8.6 Quantization for serving

Every byte a decode step reads from HBM eats bandwidth, so shrinking the number of bytes speeds decode up directly. Serving-time quantization comes in three flavors, applied to three different objects.

Weight-only quantization keeps the compute in higher precision (BF16 or FP16) and stores the weights in 4 bits. GPTQ (Frantar et al., ICLR 2023) is the post-training layer-wise round-to-nearest with error correction that shipped everywhere first; AWQ (Lin et al., MLSys 2024) is the activation-aware variant that protects the ~1% of weight channels the activations most rely on. Both target INT4 (or NF4 for QLoRA-style flows) with typical accuracy drops under one point on standard benchmarks.

Weight-and-activation quantization puts both the weights and the activations at low precision so the matmul itself runs on the low-precision tensor cores. SmoothQuant (Xiao et al., ICML 2023) and its successors handle the activation-outlier problem by migrating scale from activations into weights. On Hopper this typically means FP8; the DeepSeek-V3 report is the reference for a full FP8 training and serving stack.

KV-cache quantization is applied on top of the model weights. FP8 or INT8 KV cache typically loses less than a point on long-context evals while cutting the cache footprint in half, which doubles the batch size a GPU can hold under load. Every major serving engine now supports this as a separate flag from model quantization.

8.7 Disaggregated prefill and decode

Prefill and decode fight each other for the same GPU. Prefill wants to be interrupted as little as possible so its long matmuls run to completion; decode wants tight latency on each token. Put them on the same box and you either delay decode when a big prefill lands, or you interrupt prefill and eat context switches.

The 2023-24 answer was disaggregation: run prefill and decode on separate GPU pools, and ship the KV cache from the prefill pool to the decode pool once the prompt is done. Splitwise (Patel et al., Microsoft/UW, Nov 2023) characterized the split and prototyped the transfer. DistServe (Zhong et al., OSDI 2024) optimized the assignment jointly with SLOs. Mooncake (Qin et al., Kimi/Moonshot AI, June 2024) is the production system Kimi runs on: KV cache is treated as a first-class distributed object across a CPU/GPU/SSD memory hierarchy, with a global scheduler steering requests to whichever decode node already holds the largest usable prefix. All three of the major open serving engines (vLLM, SGLang, TensorRT-LLM) have shipped disaggregated modes since.

8.8 MoE serving and expert parallelism

Mixture-of-Experts models (taught in Chapter 2) route each token to a small subset of many experts. That is a training-side story about parameters versus compute. At serving time it becomes a memory-and-communication story. DeepSeek-V3, at 671B total / 37B active, does not fit on a single node no matter how you quantize; its experts have to live spread across many GPUs.

The standard answer is expert parallelism (EP): each GPU holds a shard of the experts, and each layer's routing is followed by an all-to-all that sends every token to whichever GPU holds the experts it was routed to, then a second all-to-all to bring the results back. The choreography is bandwidth-hungry (a single decode step now involves cross-node communication), and the payoff only appears at scale. DeepSeek's V3 technical report documents a serving layout that runs prefill under 32-way expert parallel and decode under 320-way expert parallel: one expert per GPU across 40 nodes, with a second set of GPUs hosting the shared and redundant experts. Smaller deployments trade latency for less all-to-all. The open-source DeepEP kernels released in early 2025 gave the community a reference implementation.

8.9 What reasoning did to the economics

Every trick above was in flight by early 2024, when the median request was a chat turn of maybe a few hundred output tokens. Then o1 shipped in September 2024 and the median request under reasoning models suddenly generated tens of thousands of tokens of hidden chain-of-thought before answering. That was a step-function change in the workload the engines had to serve.

Three effects fell out immediately. First, decode-time dominates the request: prefill on a short question is dwarfed by the reasoning trace that follows, which pushed even more priority onto continuous batching and speculative decoding for the long tail. Second, the KV cache per request grew by an order of magnitude, which is exactly why MLA and heavy cache quantization went from nice-to-have to load-bearing. Third, the product surface changed. Every major provider now exposes a thinking budget knob on the reasoning models, a way to bound how much compute you buy for a single response.

• OpenAI's o-family and GPT-5+ accept a reasoning.effort field on the request (from minimal through high, varying by model), which caps hidden-thinking tokens.
• Anthropic's Claude reasoning API exposes extended thinking: an explicit integer budget_tokens on the Claude 4-era models, and an adaptive-thinking mode with an effort knob on the Claude 5 line. Thinking tokens are billed at the output-token rate and reported back in usage.
• Google's Gemini 2.5 / 3.x API exposes a thinking_level knob with the same shape.

The knob is the API-visible shadow of everything in this chapter. If reasoning tokens were free, no one would need to bound them; they are not, because they arrive at bandwidth-bound decode rate, one token at a time, and every one of them holds a slot in the KV cache. The pricing sheet is now more legible than in 2023. Most providers publish separate input, cached input, and output rates, and the output rate is typically 3-5× the input rate. That gap is exactly the cost of decode.

8.10 What this chapter changed

A frontier model in 2026 is two very different bills. Training is a one-off capital spend; inference is a per-token operating cost, and reasoning-model generations pushed the operating cost into places it had never been. The plumbing that emerged to absorb that push (paged and prefix-cached KV, continuous batching, speculative decoding with EAGLE- or MTP-style drafters, FP8 weights and activations, per-request KV quantization, and disaggregated prefill/decode with expert parallelism for MoE serving) is what makes a $0.60-per-million-input-token reasoning API mathematically possible.

The tokens are cheap now, which changes what you can build on top of them. Chapter 9 picks up the moment a chat turn stops being the atomic unit: an agent spends its budget in a loop, calling tools, reading their output, and deciding what to do next. Every trick in this chapter matters more when the same model is called dozens of times inside one user request.