Chapter 4 · 2023 – 2026 · lever: democratization
The recipe gets cheaper
RLHF worked, but the recipe was expensive. From 2023 through 2026 it was taken apart in two directions at once. The alignment algorithm shed pieces - DPO dropped the reward model and the RL loop, its variants pruned it further, GRPO dropped the critic. The base model stopped being a secret - LLaMA leaked the weights into the open, and Alpaca and Vicuna showed you could fine-tune a respectable assistant on a credit card. The compute stopped being a wall - LoRA, then QLoRA, fit a 65B fine-tune onto a single GPU. Same destination as Chapter 3, an order of magnitude cheaper at every step. And then, once verifiable rewards arrived at the end of 2024, the field walked half of that back: strong open recipes returned to online RL, and the open-weights wave rolled forward through Kimi K2 and gpt-oss.
The Chapter 3 recipe - supervised fine-tuning, then a reward model, then PPO - works, and ChatGPT was the proof. But running it required keeping four models resident at once (policy, reference, reward, critic), and PPO is notoriously finicky to tune. Most of the next two years is the same field removing moving parts from that recipe, while a parallel democratization wave brings the base models and the compute within reach of anyone with a single GPU and a weekend.
4.1DPO - your LM is secretly a reward model
The DPO paper, posted to arXiv in May 2023 and a NeurIPS 2023 Outstanding Main Track Runner-Up, has the best title in the field - "Your Language Model is Secretly a Reward Model." The idea: under the KL-constrained RL objective that RLHF actually optimizes, the optimal policy is related to the reference policy through the reward in closed form. Invert that mapping and the reward function disappears into the policy itself. You can train directly on pairwise preference data - a "chosen" answer and a "rejected" answer - with a single classification loss. No reward model. No PPO rollouts. No critic. (Rafailov et al., 2023; NeurIPS award.)
Given a chosen response $y_w$ and a rejected one $y_l$ for prompt $x$, maximize the log-likelihood gap against a frozen reference policy:
$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left( \beta \log \htmlData{tip=how much more likely the preferred answer is under the trained model vs the frozen reference}{\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right)$
No reward network appears in this loss. The Bradley–Terry preference model has been folded directly into the policy, and the KL anchor to the reference policy survives as the $\beta$ coefficient.
In practice DPO is stable where PPO is twitchy, runs without an online sampling loop, and matches or beats PPO on the standard benchmarks. It also lowered the barrier: you no longer needed a reward-modeling infrastructure to ship an aligned model. A small team with a dataset of preference pairs and an SFT checkpoint could now do alignment with a regular supervised-learning workflow.
4.2DPO's family
Once the closed-form derivation was public, the year that followed produced a small zoo of DPO variants. Each one keeps the "no reward model, no PPO loop" idea and moves one other piece. It is worth knowing the shape of the space; you will see these names in almost every 2024-2025 post-training report.
| Method | Date | What it drops or changes vs DPO |
|---|---|---|
| IPO (Azar 2023) | Oct 2023 | Replaces the Bradley-Terry log-sigmoid with an identity mapping, so the loss stops running off to infinity on near-certain preferences. Same setup, less overfit. |
| KTO (Ethayarajh 2024) | Feb 2024 | No pairs. Each example is a single response tagged desirable or undesirable, and the loss is a Kahneman-Tversky utility. Match the data you actually have (a thumbs-up feed) instead of the data DPO wants (ranked A/B). |
| ORPO (Hong 2024) | Mar 2024 | Folds SFT and preference optimization into a single objective with an odds-ratio term. No reference model, no separate SFT run. One stage instead of two. |
| SimPO (Meng 2024) | May 2024 | Reference-free too, with a length-normalized log-prob as the reward and a target margin. Half the memory of DPO and reported gains on AlpacaEval / Arena-Hard. |
The moral of the family is that DPO is one point in a small design space. What you drop - the reference policy, the pair structure, the log-sigmoid, the separate SFT stage - is up to your data and your budget. What you keep is the core move: no reward model, no online rollouts.
4.3GRPO - the group is its own baseline
DPO took alignment out of reinforcement learning. GRPO stayed in, and shed something different. Introduced by DeepSeek in February 2024 inside the DeepSeekMath paper, GRPO drops the PPO critic - the value head that estimates expected return. The critic is roughly as big as the policy and has to be held in memory alongside it; killing it frees a significant chunk of compute. (Shao et al., 2024; PPO baseline: Schulman et al., 2017.)
The trick is the name: group-relative. For each prompt, sample a group of $G$ candidate responses, score all of them with the reward model, and use the group's own statistics as the baseline. Each response's advantage is just how far its reward sits above the group mean, in units of the group's standard deviation:
$\hat{A}_{i,t} = \dfrac{r_i - \htmlData{tip=average reward across the G responses sampled for this prompt}{\text{mean}(r_1,\dots,r_G)}}{\text{std}(r_1,\dots,r_G)}$
The group of $G$ samples sharing a prompt is the baseline. No critic network needed.
That's the whole change. Policy and reference model still live in memory. Reward model still scores. PPO's clipped surrogate update still applies. Only the critic - a second model-sized set of activations and optimizer state - is gone. The same GRPO comes back in Chapter 6: it is the algorithm DeepSeek uses to train R1, the open reasoning model that rivals o1. GRPO also, it turns out, has its own failure modes - length bias, entropy collapse, off-policy drift under long rollouts - and the 2025 successors (Dr. GRPO, DAPO, GSPO, CISPO, VAPO) fix them one by one. That whole zoo is Chapter 7's subject; here it is enough to know GRPO was the on-ramp.
Three recipes, side by side
The story across this chapter so far is one of progressive simplification. PPO holds four models in memory. DPO needs two and no RL loop. GRPO needs the reward signal but drops the critic.
| PPO | DPO | GRPO | |
|---|---|---|---|
| Reward model needed? | Yes | No | Yes |
| Critic / value head? | Yes | No | No |
| On-policy sampling? | Yes | No - static pairs | Yes - group of $G$ |
| Baseline for advantage | Learned critic | (no RL update) | Group mean / std |
Two different ways to shrink the same recipe - and a hint of what's coming. DPO showed you could align without RL at all. GRPO showed you could keep RL but drop the most expensive piece. By Chapter 6 the second lane reaches its logical end: reasoning models trained with verifiable rewards, where the human-labeled reward model is gone too.
4.4What happened next: back to online RL
The neat DPO-vs-GRPO framing above is where the story stood in late 2024. What actually happened over the following year is more interesting: the open community walked DPO's dominance partly back. Once verifiable rewards for math and code showed up as a reliable reward signal, the frontier of open post-training shifted the last stage from an offline preference loss to an online RL loop again - not the human-labeled PPO of 2022, but a GRPO-family run against automatic verifiers.
The clearest inflection is Ai2's Tulu 3 in November 2024. Tulu 3 published a four-stage open recipe - data curation, SFT, preference optimization (DPO), and a final stage the paper introduces as "a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)." The term RLVR was coined here. The recipe hit or exceeded closed-model baselines like Claude 3.5 Haiku and GPT-4o-mini at its scale, and the paper's headline result is the clean SFT → DPO → RLVR pipeline that most open post-training this year still starts from. (Lambert et al., 2024; Ai2 report.)
Once verifiable rewards worked, everything downstream leaned back into online RL. DeepSeek-R1 (January 2025) built on GRPO with rule-based rewards. The 2025 successor zoo - DAPO, Dr. GRPO, GSPO, CISPO, VAPO - all live in the online-RL lane. Kimi K2's post-training (July 2025) is on-policy RL over verifiable and self-critique signals. Qwen3's post-training (April 2025) is a four-stage pipeline that ends in RL over verifiable rewards, then general rewards, in that order. The DPO-only frontier lasted about a year. Chapter 7 opens exactly on that pivot. (DeepSeek-R1 Guo et al., 2025; Qwen3 Yang et al., 2025; Kimi K2 Team, 2025.)
None of this makes DPO obsolete. As the middle stage of a modern open recipe (SFT → DPO → RLVR), it still does the load-bearing preference work on tasks without cheap verifiers - style, safety, tone, refusals. And on a small team's budget it is still the simplest thing that works. DPO's story is not "beaten" - it is "sandwiched" between an SFT stage that teaches format and an RL stage that chases the things a verifier can actually measure.
4.5The open-weights wave
The other half of this chapter isn't about algorithms. It's about who gets to run them. In February 2023, Meta released LLaMA - seven, thirteen, thirty-three, and sixty-five-billion-parameter base models under a non-commercial research license, available to approved researchers. The headline claim in the paper was already a shock: LLaMA-13B outperforms the 175B GPT-3 on most benchmarks, and the 65B sits in the same league as Chinchilla-70B and PaLM-540B. A 13B model wasn't supposed to be able to do that. (Touvron et al., 2023; Meta announcement.)
A week later, on 3 March 2023, the weights leaked. Someone posted a torrent on 4chan, the files propagated through HuggingFace pull requests, and by the time Meta filed DMCA takedowns the leak was already everywhere. From that moment forward, "open-weights LLaMA derivatives" were a fact on the ground regardless of the license. (Timeline.) When Meta released LLaMA 2 in July 2023 under a commercial-permissive license - with one carve-out for licensees above 700M monthly active users - the open-weights wave stopped being gray-zone leakage and became official Meta strategy. (LLaMA 2 license.)
What happened in the weeks after the LLaMA-1 leak set the template for everything since.
Alpaca (Stanford CRFM, 13 March).
Take LLaMA-7B, generate 52K
instruction-following examples
by self-instruct from text-davinci-003, fine-tune. Cost: under $600 total
(~$500 in OpenAI API calls plus <$100 in cloud compute). In a blind pairwise
evaluation, Alpaca tied text-davinci-003, 90 wins to 89.
(CRFM.)
Vicuna (LMSYS, 30 March). Take LLaMA, fine-tune on ~70K real ChatGPT conversations from ShareGPT. LMSYS reported ~$300 for the 13B run and used GPT-4 as a judge to call Vicuna-13B "more than 90% of the quality of ChatGPT and Bard." Their own blog post flagged the eval as "fun and non-scientific" - a useful reminder that an LLM judge can be Goodharted long before the underlying model actually catches up. (LMSYS.)
The headline numbers in those posts aged unevenly, but the shape of the moment is what mattered: in two weeks, a research community with no access to GPT-4-scale training infrastructure had produced credible instruction-tuned assistants for a few hundred dollars each, on top of a base model someone else had paid millions to train. The cost floor of "make a useful chat model" had dropped two orders of magnitude in a fortnight.
The three years since bear the same signature at bigger scale. The base models have gotten stronger and more open, and by 2025 the frontier of "open weights" is indistinguishable from the frontier of "closed weights" on most public benchmarks.
| When | Release | Why it mattered |
|---|---|---|
| Sep 2023 | Mistral 7B | First Apache-2 base that beat LLaMA-2-13B at 7B. The permissive-license bar reset. |
| Dec 2023 | Mixtral 8x7B | The open MoE that made "sparse experts" a household concept for practitioners. |
| Jul 2024 | Llama 3.1 405B | First open ~GPT-4-tier flagship (the herd's 8B and 70B, also over-Chinchilla at ~15T tokens - the deliberate over-training move from Ch 1). |
| Dec 2024 | DeepSeek-V3 | 671B MoE with 37B active, aux-loss-free routing, FP8 training. The recipe that everyone in the open community read carefully. |
| Jan 2025 | DeepSeek-R1 | Open reasoning model that rivalled o1 on math and code, trained mostly by RL. This is the file Chapter 6 opens on. |
| Apr 2025 | Qwen3 | Apache-2 dense and MoE lines from 0.6B to 235B. Hybrid thinking / non-thinking modes in one model, 119 languages. |
| Jul 2025 | Kimi K2 | 1T-param MoE with 32B active; modified-MIT license; the open flagship of the summer. |
| Aug 2025 | gpt-oss-120b / 20b | OpenAI's first open-weights release since GPT-2. Apache-2 MoE, the 120B near o4-mini, fits on one 80 GB H100. |
| Aug 2025 | DeepSeek-V3.1 | Hybrid think / non-think in one model, 128K context, still open. |
| Dec 2025 | DeepSeek-V3.2 + V3.2-Speciale | V3.2 upgraded both deepseek-chat (non-think) and deepseek-reasoner (think), alongside a time-limited V3.2-Speciale endpoint at V3.2 pricing. |
| Apr 2026 | DeepSeek-V4 (Pro + Flash) | The V3 line closes; V4 opens with the same open recipe plus a thinking-effort dial (low/high/max) at the API layer. |
| Jul 2026 | Kimi K3 | 2.8T-parameter Moonshot flagship on Kimi Delta Attention, 1M-token context; the open-weight coding-and-agent option of the summer. Custom license, not permissive: revenue-share plus a required contract with Moonshot for any provider above US$20M annual revenue. |
| Aug 2026 | Qwen3.8-Max + Qwen3.8-27B | 2.4T sparse-MoE Alibaba flagship (proprietary API tier) plus a 27B Apache-2 checkpoint two weeks later; the open cut still ships alongside the closed one. |
| Sep 2026 | DeepSeek-V4.1-Flash | 552B asymmetric causal encoder / decoder MoE (8B in, 16B out active); a quarter of the HBM and an eighth of the SSD of V4-Flash for KV cache, at a fraction of the price. |
The through-line is that the open-weights wave stopped being a research toy and became the plurality of the frontier. A working ML engineer in late 2026 can pick a fully open MoE flagship, fine-tune it against a written spec, and serve it - without a phone call to a research lab. That is the point on the map where the "fine-tune it in my voice" idea from the intro has both the base model and the recipe within reach.
4.6PEFT - the compute wall comes down
Alpaca and Vicuna were still full fine-tunes, which is why they leaned on the 7B and 13B sizes. To touch 65B you needed a small cluster. The fix was already waiting: parameter-efficient fine-tuning - freeze the pretrained weights and train a small adapter beside them.
LoRA (Hu et al., June 2021) is the version that won. The pretrained weight matrix $W$ stays frozen; the update is constrained to a low-rank factorization $\Delta W = BA$, where $A \in \mathbb{R}^{r\times k}$, $B \in \mathbb{R}^{d \times r}$, and the rank $r$ is much smaller than $\min(d, k)$. Train only $A$ and $B$. At inference, merge $BA$ back into $W$ - no extra latency. Vs full Adam fine-tuning of GPT-3 175B, the paper reports ~10,000× fewer trainable parameters and ~3× less GPU memory, with no quality drop. (Hu et al., 2021.)
QLoRA (Dettmers et al., May 2023) took the next step: keep LoRA's frozen-base / trainable-adapter idea, but also quantize the frozen base to 4-bit. Three new tricks made it work without degradation. NF4 - a 4-bit NormalFloat data type, information-theoretically optimal for normally distributed weights. Double quantization - quantizing the quantization constants themselves, saving another ~0.4 bits per parameter on average. Paged optimizers using NVIDIA unified memory to spill optimizer state to CPU during the gradient spikes that would otherwise OOM. The payoff: fine-tuning a 65B model on a single 48 GB GPU, 24 hours of training, matching 16-bit full fine-tuning quality. The Guanaco family that came out of this hit 99.3% of ChatGPT on the Vicuna benchmark. (Dettmers et al., 2023.)
If you've been ML-literate since 2019 and you've been scoping a "fine-tune a small model on my own writing in my own voice" project, this is exactly the node where that plan lives. QLoRA on a single consumer-ish GPU was the credible 2023-vintage path to personal fine-tuning, and it is a real one. Hold onto it - the Chapter 11 capstone returns to it and argues, with a course's worth of evidence, that for style specifically the prompt lane (Chapter 5) is the better path. Plant the seed; we'll resolve it.
4.7The democratization curve, on one axis
Notice that the two halves of this chapter are the same curve seen from two angles.
| Lever | 2022 baseline | 2023 | 2024 | 2025 - 2026 |
|---|---|---|---|---|
| Alignment algorithm | PPO (4 models) | DPO (2 models, no RL) | GRPO + Tulu 3's SFT→DPO→RLVR | GRPO successors (Ch 7) |
| Base model access | API-only | LLaMA → Alpaca / Vicuna | LLaMA-2 → Mixtral → DeepSeek-V3 | Qwen3, Kimi K2, gpt-oss |
| Compute footprint | Full FT (cluster) | LoRA (single GPU) | QLoRA (48 GB) | Same, on open MoE bases |
Every row is the same gesture: keep what you're trying to do, drop something the previous generation needed. None of this changed the shape of the alignment recipe Chapter 3 laid out. Better algorithms, more open weights, smaller GPUs - the map is just being filled in cheaper.
4.8What this chapter didn't touch
Two things, on purpose. The PEFT family is wider than just LoRA and QLoRA - adapters, prefix-tuning, IA³, others - and they all live in the weights lane. The PEFT survey is the place to go if you want the taxonomy. (Han et al., 2024.)
And note what every method in this chapter still does: it touches the weights. DPO, GRPO, LoRA, QLoRA - cheaper at every step, but still gradient descent into parameters. The next chapter asks a stranger question. What if you never touch the weights at all, freeze the model entirely, and treat the prompt as the thing you optimize? That move turns out to rebuild this entire RL machinery out of language.