From Language Models to Assistants

Large language models are trained to predict the next token in a sequence. This objective produces models with remarkable breadth of knowledge—they absorb trivia, syntax, coreference patterns, sentiment, and even rudimentary reasoning from the vast text corpora they consume. Yet next-token prediction alone does not produce a helpful assistant. A model trained purely on web text will happily continue any prompt, but it has no notion of what a user actually wants.

The gap between "predict the next token" and "be a useful assistant" is substantial. Bridging it requires a sequence of interventions: first, instruction finetuning teaches the model to follow directions; then, preference optimization (via RLHF or DPO) aligns its outputs with human judgments of quality. This lecture focuses on the second stage—the techniques that transform a capable but unaligned model into one whose responses humans actually prefer.

What Pretraining Learns

Pretraining on next-token prediction yields a model that captures diverse linguistic and world knowledge. Given "Stanford University is located in __________, California," a pretrained model fills in "Stanford" without hesitation. It handles syntax ("I put ___ fork down" → "my"), coreference ("The woman walked across the street, checking for traffic over ___ shoulder" → "her"), and lexical semantics ("fish, turtles, seals, and ___" → "whales"). It can even perform simple sentiment analysis and basic arithmetic, though more complex reasoning remains fragile.

Language models also function as rudimentary world models. They can serve as agents (generating actions in interactive environments), solve math problems (Khan Academy-style), write code (GitHub Copilot), and provide medical information. The common thread is that pretraining captures patterns in how the world works—but only insofar as those patterns are reflected in text.

Instruction Finetuning

The first step toward building an assistant is instruction finetuning—training the model on (instruction, output) pairs across many tasks. The idea is straightforward: collect examples of desired behavior, then maximize the likelihood of the correct output conditioned on the instruction. This is the same supervised learning objective used in pretraining, but now the data consists of explicit task demonstrations rather than raw text.

FLAN-T5 (Chung et al., 2022) demonstrated the power of this approach at scale: T5 encoder-decoder models finetuned on 1,800 additional tasks showed dramatic improvements on held-out benchmarks like BIG-bench and MMLU. The gains were especially pronounced for larger models—bigger models benefited more from instruction finetuning, suggesting that the base model's capacity determines how much alignment signal it can absorb.

Limitations of Instruction Finetuning

Despite its effectiveness, instruction finetuning has three important limitations:

• No right answer for open-ended tasks. For creative writing, open-ended dialogue, or stylistic preferences, there is no single ground-truth output. "Write me a story about a dog and her pet grasshopper" has infinitely many good completions, and standard supervised learning cannot express which ones are better than others.
• Equal penalty for all errors. Language modeling penalizes every incorrect token equally, but some mistakes are far more consequential than others. Getting a factual claim wrong is much worse than choosing a slightly different synonym.
• Suboptimal human demonstrations. Humans generate imperfect answers. Even the best annotators produce responses that could be improved, and the model learns to imitate these imperfections faithfully.

The core issue is a mismatch between the language modeling objective (maximize token-level likelihood) and the true goal (satisfy human preferences). We need an optimization target that directly captures what humans prefer.

The RLHF Pipeline

Reinforcement Learning from Human Feedback (RLHF) addresses this mismatch head-on. Instead of maximizing likelihood against a fixed target, we optimize the model to produce outputs that receive high reward according to a learned model of human preferences. The full pipeline consists of three stages.

Defining the Reward Signal

Suppose we have an instruction $x$ and a model-generated response $y$. We want a scalar reward $R(x, y) \in \R$ indicating how good the response is—higher is better. If we had access to reliable human ratings, we could directly maximize:

But obtaining direct human reward scores for every model output is prohibitively expensive. Even if we could, human ratings are noisy and poorly calibrated—different annotators assign wildly different absolute scores to the same response.

Pairwise Comparisons and the Bradley-Terry Model

The solution is to collect pairwise comparisons rather than absolute ratings. Given two responses $y_w$ (preferred, or "winning") and $y_l$ (dispreferred, or "losing") to the same prompt $x$, a human annotator simply indicates which is better. Comparisons are more reliable than absolute scores because they bypass calibration issues.

To train a reward model from comparison data, we use the Bradley-Terry paired comparison model (Bradley and Terry, 1952). Under this model, the probability that response $y_w$ is preferred over $y_l$ is:

where $\sigma$ is the logistic sigmoid function. The reward model $RM_\phi(x, y)$ is typically a transformer that takes the prompt-response pair as input and produces a scalar score. We train it by minimizing the negative log-likelihood of the observed preferences:

KL-Constrained Policy Optimization

With a trained reward model $RM_\phi$ in hand, we now want to optimize the language model's parameters $\theta$ to maximize the predicted reward. A naive approach would be to simply maximize $\E_{\hat{y} \sim p_\theta^{RL}(\cdot \mid x)}[RM_\phi(x, \hat{y})]$, but this leads to reward hacking—the model finds degenerate outputs that score highly under the (imperfect) reward model while being clearly bad by human standards.

The standard remedy is to add a KL divergence penalty that prevents the policy from straying too far from the pretrained or SFT model $p^{PT}$:

The penalty term $\beta \log \frac{p_\theta^{RL}(\hat{y} \mid x)}{p^{PT}(\hat{y} \mid x)}$ charges a price whenever the RL policy assigns substantially more probability to a response than the pretrained model does. In expectation over $\hat{y}$, this penalty is exactly $\beta \cdot \Dkl\!\left(p_\theta^{RL}(\cdot \mid x) \,\|\, p^{PT}(\cdot \mid x)\right)$, the KL divergence from the pretrained model to the RL policy.

Policy Gradient Recap

How do we actually optimize this objective? The reward model is generally not differentiable with respect to the discrete text generation process, so we cannot simply backpropagate through it. Instead, we use policy gradient methods, specifically the REINFORCE estimator (Williams, 1992).

The key mathematical tool is the log-derivative trick. Starting from the gradient of the expected reward:

We use the identity $\nabla_\theta \, p_\theta(y \mid x) = p_\theta(y \mid x) \, \nabla_\theta \log p_\theta(y \mid x)$ (which follows from the chain rule applied to $\log p_\theta$) to rewrite this as an expectation:

This can be approximated via Monte Carlo sampling: draw $m$ samples $y_1, \ldots, y_m$ from $p_\theta$ and compute:

The intuition is straightforward: if the reward $R(x, y_i)$ is large and positive, take a gradient step that increases $\log p_\theta(y_i \mid x)$—making this good response more likely. If the reward is negative, decrease its probability. This is why the method is called "reinforcement" learning: good actions are reinforced.

Direct Preference Optimization (DPO)

The RLHF pipeline works, but it is complex. It requires training a separate reward model, running on-policy sampling during RL, fitting a value function for variance reduction, and carefully tuning many hyperparameters. Rafailov et al. (2023) showed that this entire pipeline can be collapsed into a single supervised learning objective by exploiting the closed-form solution to the KL-constrained optimization problem.

Closed-Form Solution to the KL-Constrained Problem

Recall the RLHF objective for a fixed prompt $x$:

This is a constrained optimization problem over the distribution $p$. Writing out the KL divergence explicitly and taking the functional derivative with respect to $p(y \mid x)$, one can show that the optimal policy takes the form:

The Implicit Reward Parameterization

The closed-form solution above tells us the shape of the optimal policy given a reward function. The key DPO insight is to rearrange this expression to extract the reward in terms of the policy. Taking logs of both sides:

Solving for the reward:

This is remarkable: it expresses the reward function entirely in terms of the optimal policy $p^*$ and the reference model $p^{PT}$, plus a prompt-dependent constant $\beta \log Z(x)$. Since this relationship holds for any policy (not just the true optimal one), we can parameterize the implicit reward using our trainable policy $p_\theta^{RL}$:

The DPO Loss Function

Now we substitute this implicit reward into the Bradley-Terry preference model. Recall that the reward model training objective depends only on the difference in rewards between the preferred and dispreferred responses:

The partition function $\beta \log Z(x)$ cancels out, since it depends only on the prompt, not on the response. Substituting into the Bradley-Terry loss gives the DPO objective:

DPO elegantly collapses the three-stage RLHF pipeline (SFT, reward model training, RL optimization) into a single supervised learning step. The only inputs needed are a reference model $p^{PT}$, preference pairs $(x, y_w, y_l)$, and standard gradient descent. There is no separate reward model, no on-policy sampling, and no value function to fit.

RLHF versus DPO: Trade-offs

Both RLHF and DPO optimize the same underlying objective—KL-constrained reward maximization from preference data—but they differ substantially in implementation.

Strengths of RLHF

• Online data. RLHF generates fresh samples from the current policy during training, allowing the reward model to evaluate the model's actual outputs. This on-policy feedback can be crucial when the distribution of outputs shifts significantly during training.
• Proven at scale. InstructGPT (Ouyang et al., 2022) scaled RLHF to tens of thousands of tasks and demonstrated substantial improvements in helpfulness, truthfulness, and harmlessness. ChatGPT built on this approach for dialogue agents.
• Flexibility. The explicit reward model can be reused across multiple training runs, analyzed for failure modes, and combined with additional reward signals.

Strengths of DPO

• Simplicity. No reward model, no value function, no on-policy sampling. DPO is a single supervised learning pass over preference data.
• Stability. RL optimization is notoriously sensitive to hyperparameters. DPO avoids the instabilities of policy gradient training—there is no reward hacking, no mode collapse from aggressive optimization, and no need to tune PPO-specific parameters.
• Computational efficiency. DPO requires only forward and backward passes through the policy model (and a frozen reference model for computing $p^{PT}$), avoiding the expensive sampling loop of RL.

DPO has been widely adopted in the open-source LLM ecosystem, where most models now use DPO (or variants) as part of their alignment pipeline. However, DPO's reliance on offline preference data means it cannot explore new parts of the output space during training, which can limit performance when the preference dataset does not cover the model's current behavior.

InstructGPT and ChatGPT

The most influential application of RLHF at scale was InstructGPT (Ouyang et al., 2022), which applied the full three-stage pipeline to GPT-3. The process began with supervised finetuning on 13,000 demonstration examples, followed by reward model training on 33,000 human comparisons, and finally PPO-based RL optimization. The result was a model that dramatically outperformed the much larger base GPT-3 on human preference evaluations—the 1.3B parameter InstructGPT was preferred over the 175B GPT-3, demonstrating that alignment can substitute for raw scale.

InstructGPT covered approximately 30,000 tasks spanning generation, question answering, dialogue, summarization, classification, and more. The diversity of task types was essential: the model needed to generalize the "be helpful, harmless, and honest" signal across widely varying contexts.

ChatGPT extended this approach to multi-turn dialogue. While OpenAI disclosed fewer training details, the core recipe remained the same: instruction finetuning followed by RLHF, with the dialogue format requiring careful handling of conversation history and turn-taking. The resulting model exhibited clear stylistic changes compared to the base model—responses became more detailed, better formatted, and structured with lists and headers (Dubois et al., 2023).

Limitations of Preference Optimization

Both RLHF and DPO share fundamental limitations that stem from the difficulty of specifying and modeling human preferences.

Reward Hacking

Reward hacking occurs when the model finds ways to achieve high predicted reward without actually satisfying human preferences. Because the reward model is an imperfect proxy for human judgment, sufficiently aggressive optimization will exploit its errors. OpenAI documented dramatic examples: models trained with RL sometimes produce degenerate outputs that score highly under the reward model but are clearly nonsensical to humans.

The KL constraint mitigates this by limiting how far the policy can deviate from the reference model, but it does not eliminate the problem. In practice, there is a characteristic pattern of reward model over-optimization: as the KL divergence from the reference model increases, the true (human-evaluated) quality initially improves but eventually degrades, even as the predicted reward continues to rise (Stiennon et al., 2020).

Hallucinations and Sycophancy

RLHF-trained chatbots are rewarded for producing responses that seem authoritative and helpful. This creates an incentive to state claims confidently, regardless of whether they are true—leading to hallucinations (making up facts) and sycophancy (telling users what they want to hear rather than what is correct). These failure modes are difficult to address through preference optimization alone, because human annotators often cannot distinguish confident-sounding falsehoods from truth.

Data Expense

Collecting high-quality human preference data remains expensive. Each comparison requires a trained annotator to read and evaluate two full model responses, and annotator agreement rates are often modest. This cost limits the scale and coverage of preference datasets, particularly for specialized domains.

Future Directions

Several lines of research aim to address the limitations of current preference optimization methods.

RL from AI Feedback (RLAIF)

Constitutional AI (Bai et al., 2022) replaces human annotators with AI-generated feedback. The model critiques its own outputs according to a set of principles (a "constitution"), then revises its responses. The critique-revision pairs serve as training data for preference optimization. This dramatically reduces the cost of data collection while allowing the alignment criteria to be specified declaratively rather than implicitly through annotator behavior.

Self-Improvement and Self-Taught Reasoning

Another direction is to finetune language models on their own successful outputs. The Self-Taught Reasoner (STaR) approach (Zelikman et al., 2022) works as follows: the model generates chain-of-thought reasoning traces for a set of problems, filters for those that produce correct answers, and then trains on the successful traces. Over multiple iterations, the model bootstraps increasingly sophisticated reasoning abilities from its own generations. This is particularly effective for domains like code and mathematics where correctness can be verified automatically.

Personalization

Current preference optimization treats "human preferences" as a monolithic signal, but in reality preferences vary enormously across individuals. The PRISM Alignment Project (Kirk et al., 2024) and Few-Shot Preference Optimization (FSPO) (Singh and Hsu et al., 2024) explore how to adapt language models to individual users' preferences without requiring per-user RLHF. This is an increasingly important direction as language models are deployed to diverse user populations with varying needs, values, and communication styles.

Summary

Preference optimization transforms capable but unaligned language models into helpful assistants. The journey from pretraining through instruction finetuning to RLHF or DPO represents a progression from learning what text looks like to learning what users want.

The core mathematical insight underlying both RLHF and DPO is the KL-constrained reward maximization problem. RLHF solves it via policy gradient methods applied to a learned reward model; DPO exploits the closed-form solution to bypass the reward model entirely, reducing preference alignment to a binary classification problem. Both approaches have been validated at scale—InstructGPT and ChatGPT for RLHF, and a growing number of open-source models for DPO.

The limitations of preference optimization—reward hacking, hallucinations, sycophancy, and data cost—remain active research challenges. Emerging approaches like Constitutional AI, self-taught reasoning, and personalized alignment offer promising paths forward, but the fundamental tension between optimizing for appearing good and being good continues to define the frontier of the field.