Lecture 8

Reward Learning

When reward functions are hard to specify: learning from human preferences, RLHF, and inverse reinforcement learning.

RLHF Inverse RL Preference Learning Reward Modeling
Original PDF slides

Where Does the Reward Come From?

Every reinforcement learning algorithm we have studied so far takes the reward function as given. In Atari games, the reward is the score on screen. In simulated robotics, the reward is a hand-coded function measuring distance to a goal. But what about real-world problems?

Consider the following domains: a robot manipulating objects in a kitchen, a dialogue agent that should be helpful and harmless, or an autonomous vehicle navigating traffic. In each case, the question "what is the reward?" has no obvious answer. The true objective is complex, multi-faceted, and difficult to express as a mathematical function. In practice, engineers often resort to proxy rewards—simplified approximations that are easy to compute but may not capture the full intent. Optimizing a proxy reward can lead to unexpected and undesirable behaviors, a phenomenon sometimes called reward hacking.

Key Insight
The reward function is the single most important specification in an RL system—it defines what the agent should do. Yet in most real-world settings, rewards cannot be taken for granted. Task specification is hard, and getting it wrong can be catastrophic. This lecture covers methods for learning rewards from human supervision.

We have already seen one alternative to hand-designed rewards: imitation learning, where the agent mimics an expert's actions. But imitation learning has limitations—it does not reason about outcomes or dynamics, the expert may have different capabilities than the agent, and demonstrations may not always be available. Can we do better?

Learning Rewards from Goal Examples

Perhaps the simplest form of reward learning is to learn a goal classifier—a binary classifier that distinguishes goal states from non-goal states, whose output serves as a reward signal.

The Basic Approach

The idea is straightforward:

Goal Classifier as Reward
  1. Collect examples of successful states (inside goal set $G$) and unsuccessful states (outside $G$).
  2. Train a binary classifier $C_\psi(s)$ with inputs $s_i$ and labels $\mathbf{1}(s_i \in G)$.
  3. Run RL with $C_\psi(s)$ as the reward function.
Example
Robotic Manipulation. Consider a task where a robot must place a pencil case behind a notebook. We collect images of successful arrangements (positive examples) and unsuccessful ones (negative examples), train a binary classifier on these images, and use the classifier's output probability as the reward signal for RL.

The Exploitation Problem

There is a fundamental problem with using a static classifier as a reward: the RL algorithm will seek out states where the classifier outputs a high value. It may simply find states that the classifier was not trained on—effectively exploiting the classifier's weaknesses rather than achieving the actual goal. This is a form of reward hacking that is especially pernicious because neural network classifiers can be confidently wrong on out-of-distribution inputs.

Key Insight
Any learned reward function is a model, and models have errors. When an RL agent optimizes against a learned reward, it will exploit those errors. This is a recurring theme: the same distributional shift problem from offline RL appears here in a different guise. The policy visits states that the reward model was never trained on.

Adversarial Classifier Training

The solution is to update the classifier during RL, using the states the policy visits as negative examples. This prevents the RL agent from exploiting the classifier because any states it discovers will immediately be incorporated into the classifier's training data.

Adversarial Goal Classifier
  1. Collect an initial set of successful states $D_+$ and unsuccessful states $D_-$.
  2. Update the classifier using $D_+$ and $D_-$ (balance the dataset 50/50).
  3. Collect experience $s_t, a_t, \ldots$ using the current policy $\pi$.
  4. Update policy $\pi$ using the classifier-based reward.
  5. Add visited states to negatives: $D_- \leftarrow D_- \cup \{s_t\}$.

Repeat steps 2-5.

A subtle question arises: what if some of the visited states are actually successful? Won't labeling them as negatives confuse the classifier? The key observation is that as long as the batches are balanced (50% positive, 50% negative), the classifier will output $p \geq 0.5$ for truly successful states, since they appear in both the positive and negative sets. The classifier cannot be exploited, and successful states still receive a positive reward signal.

Connection to Generative Adversarial Networks

This adversarial training scheme is closely related to Generative Adversarial Networks (GANs). In a GAN, a discriminator (classifier) learns to distinguish real data from generated data, while a generator learns to produce data the discriminator thinks is real. In our setting, the classifier plays the role of the discriminator, and the RL policy plays the role of the generator—trying to reach states that the classifier thinks are goals. At convergence, the generator (policy) should match the distribution of goal states, just as a GAN generator matches the data distribution.

Example
Self-Improving Robots. Sharma, Ahmed, Ahmad, and Finn (2023) applied this approach to real-world robotic learning. Starting with 50 demonstrations, they used the final states as success examples and initialized the RL replay buffer with the demo data. The RL policy trained with a learned classifier achieved a 62% success rate, compared to only 26% success rate for direct imitation of the demonstrations. Regularization of the classifier proved important for stable training.

Strengths and Limitations

Goal classifiers provide a practical framework for task specification:

Learning Rewards from Human Preferences

Goal classifiers require examples of successful outcomes. But what if we cannot even provide those? What if the task is so nuanced that it is difficult to specify what "success" looks like, but easy to say which of two outcomes is better?

This motivates preference-based reward learning: instead of asking humans for demonstrations or goal examples, we ask them to compare pairs of behaviors and say which one they prefer. This type of relative judgment is often much easier for humans to provide than absolute evaluations or demonstrations.

Definition
Preference-Based Reward Learning. Given a set of pairwise comparisons $\{\tau_w \succ \tau_l\}$ where $\tau_w$ is preferred over $\tau_l$, learn a reward function $r_\theta(s, a)$ such that preferred trajectories have higher cumulative reward: $$\sum_{(s,a) \in \tau_w} r_\theta(s, a) > \sum_{(s,a) \in \tau_l} r_\theta(s, a)$$ Here $\tau$ can be a full trajectory or a partial segment.

The key insight is that humans are performing a classification task—deciding which trajectory is better. The reward function should be discriminative in the same way. This leads to a natural probabilistic formulation.

The Bradley-Terry Model for Preferences

To turn pairwise comparisons into a learning objective, we use the Bradley-Terry model, a classical model from the statistics of paired comparisons. The model defines the probability that trajectory $\tau_a$ is preferred over $\tau_b$ as:

$$P(\tau_a \succ \tau_b) = \sigma\!\left(r_\theta(\tau_a) - r_\theta(\tau_b)\right)$$

where $\sigma$ is the sigmoid function and $r_\theta(\tau) = \sum_{(s,a) \in \tau} r_\theta(s, a)$ is the cumulative reward of trajectory $\tau$ under the reward model.

Definition
Bradley-Terry Model. The probability that trajectory $\tau_a$ is preferred over trajectory $\tau_b$ is modeled as: $$P(\tau_a \succ \tau_b) = \frac{\exp(r_\theta(\tau_a))}{\exp(r_\theta(\tau_a)) + \exp(r_\theta(\tau_b))} = \sigma\!\left(r_\theta(\tau_a) - r_\theta(\tau_b)\right)$$ This is equivalent to logistic regression on the difference in cumulative rewards.

The training objective follows directly: maximize the log-likelihood of the observed preferences:

$$\max_\theta \; \E_{\tau_w, \tau_l}\!\left[\log \sigma\!\left(r_\theta(\tau_w) - r_\theta(\tau_l)\right)\right]$$

where $\tau_w \succ \tau_l$ indicates that $\tau_w$ was preferred by the human annotator. This is simply binary cross-entropy loss, a familiar supervised learning objective. The reward model $r_\theta$ is typically parameterized as a neural network that takes a state-action pair (or an entire trajectory) as input and outputs a scalar reward.

Derivation of the preference loss

We model human preferences as a noisy comparison process. Given two trajectories $\tau_a$ and $\tau_b$, the human prefers $\tau_a$ with probability proportional to the exponentiated total reward of $\tau_a$:

$$P(\tau_a \succ \tau_b) = \frac{\exp(r_\theta(\tau_a))}{\exp(r_\theta(\tau_a)) + \exp(r_\theta(\tau_b))}$$

Taking the log of this probability when $\tau_a = \tau_w$ (the winner):

$$\log P(\tau_w \succ \tau_l) = \log \frac{\exp(r_\theta(\tau_w))}{\exp(r_\theta(\tau_w)) + \exp(r_\theta(\tau_l))}$$ $$= r_\theta(\tau_w) - \log(\exp(r_\theta(\tau_w)) + \exp(r_\theta(\tau_l)))$$ $$= \log \sigma(r_\theta(\tau_w) - r_\theta(\tau_l))$$

where $\sigma(x) = 1/(1 + \exp(-x))$ is the sigmoid function. Maximizing this over all preference pairs gives the Bradley-Terry loss.

The Complete Reward Learning Algorithm

Putting the pieces together, the full reward learning pipeline operates as follows:

Reward Learning from Preferences
  1. Sample trajectories. Given a dataset $\{\tau_i\}$, sample batches of $k$ trajectories and present them to human annotators for ranking. (For LLMs, these $k$ trajectories all share the same prompt.)
  2. Compute rewards. Evaluate $r_\theta(\tau_1), \ldots, r_\theta(\tau_k)$ under the current reward model $r_\theta$.
  3. Compute gradients. For all $\binom{k}{2}$ pairs per batch, compute $\nabla_\theta \E_{\tau_w, \tau_l}[\log \sigma(r_\theta(\tau_w) - r_\theta(\tau_l))]$ where $\tau_w \succ \tau_l$.
  4. Update reward model. Update $\theta$ using the computed gradient.

This can be performed offline on a fixed dataset of comparisons, or in the loop of online RL, where the policy generates new trajectories for humans to compare.

An important design choice is whether reward learning happens offline (collect all preferences first, then train a reward model and use it for RL) or online (interleave preference collection with policy training). Online reward learning can be more sample-efficient because the policy generates increasingly relevant trajectories for the human to compare, but it requires human annotators to be available during training.

Example
Learning Backflips from 900 Queries. Christiano et al. (NeurIPS 2017) demonstrated that a simulated agent could learn complex behaviors like backflips using only 900 human preference queries, combined with online RL. The human simply watched pairs of short video clips and indicated which one was better. The reward model trained from these comparisons was sufficient to drive the agent to discover the desired behavior.

Reinforcement Learning from Human Feedback (RLHF)

The most impactful application of preference-based reward learning has been in aligning large language models (LLMs). The RLHF pipeline has become the standard approach for training models like ChatGPT, Claude, and Gemini to be helpful, harmless, and honest.

The Three-Stage RLHF Pipeline

Modern LLM training typically follows three stages:

RLHF Pipeline for LLMs
  1. Large-scale pre-training. Train the model on massive text corpora using next-token prediction. The resulting model has broad language capabilities but mixed quality—it can generate both excellent and terrible responses.
  2. Supervised fine-tuning (SFT). Fine-tune on a curated dataset of high-quality (prompt, response) pairs. This teaches the model the format of good responses.
  3. RL from human feedback (RLHF):
    1. Gather preference data. For each prompt $x$, sample two (or more) responses $y$ and $y'$ from the SFT model. Ask human annotators which response is better: $y \succ y'$.
    2. Train reward model. Train a reward model $r(x, y)$ that judges how good a response $y$ is for a prompt $x$, using the Bradley-Terry loss on the preference data.
    3. RL to maximize reward. Fine-tune the language model to produce responses with high reward, typically using PPO with a KL penalty against the SFT model to prevent reward hacking.

In the LLM setting, a "trajectory" is a (prompt, response) pair. The prompt plays the role of the state, and the response (a sequence of tokens) plays the role of a sequence of actions. The reward model $r(x, y)$ takes a full prompt-response pair and outputs a scalar score.

Key Insight
The power of RLHF lies in the observation that critique is easier than generation. A human may not be able to write a perfect response to a complex question, but they can usually tell which of two responses is better. This asymmetry between generation and evaluation is what makes preference-based learning practical at scale.

Beyond Human Feedback: AI Feedback

Collecting human preferences is expensive and slow. A natural question is: can we replace human annotators with AI? This leads to Reinforcement Learning from AI Feedback (RLAIF), introduced by Bai et al. (2022) in their work on Constitutional AI.

The core idea is to ask another language model to evaluate which of two responses is better, using criteria specified in a "constitution"—a set of principles like "choose the response that is less harmful" or "choose the response that is more helpful and truthful." The key insight is the same: critique is easier than generation. Even if an AI model cannot generate a perfect response, it may be able to reliably judge which of two responses better adheres to a given principle.

Example
Constitutional AI. Bai et al. (2022) showed that using AI-generated feedback to train reward models can achieve a Pareto improvement over standard RLHF on both helpfulness and harmlessness. Their "Constitutional RL" approach, combined with chain-of-thought reasoning for the evaluator, produced models that were simultaneously more helpful and more harmless than those trained with human feedback alone.

RLAIF dramatically reduces the cost of preference data collection and enables scaling to much larger datasets. However, it introduces its own challenges: the AI evaluator may have systematic biases, and its judgments are only as good as the principles it follows and the capabilities of the evaluating model.

Practical Considerations in Reward Learning

Several practical issues arise when deploying reward learning in real systems:

Reward Hacking

Just as RL agents can exploit errors in hand-designed rewards, they can exploit errors in learned rewards. The reward model is trained on a finite dataset and may not generalize perfectly to the states the optimized policy visits. Common mitigations include:

Human Consistency and Noise

Human preferences are noisy and inconsistent. Different annotators may disagree, and even the same annotator may give different judgments at different times. The Bradley-Terry model handles this gracefully through its probabilistic formulation—it models preferences as noisy observations of an underlying quality score. However, systematic biases (e.g., preferring longer responses regardless of quality) can be problematic and require careful data collection protocols.

Active Preference Queries

Not all trajectory pairs are equally informative. Active preference learning (Sadigh et al., RSS 2017) selects the most informative pairs to present to human annotators, reducing the total number of queries needed. The most informative pairs are typically those where the current reward model is most uncertain about which trajectory is better.

Can RL Agents Propose Their Own Goals?

Beyond learning rewards from human supervision, an emerging research direction asks whether RL agents can discover their own goals through unsupervised RL. One compelling approach formulates a two-player game between a goal-setter (Alice) and a goal-reacher (Bob).

Alice proposes goals that she believes Bob cannot achieve, and Bob tries to achieve them. Through this adversarial self-play, Bob develops increasingly general skills without any external reward signal. These pre-trained skills can then be rapidly adapted to downstream tasks with minimal supervision.

Example
Asymmetric Self-Play. Sukhbaatar et al. (ICLR 2018) demonstrated this approach in a grid-world environment. Alice navigates through rooms and challenges Bob to reach her final position. This forces Bob to learn general navigation skills—including key-finding and door-opening—without any task-specific reward. When later given a target task, Bob can leverage these pre-trained skills for rapid learning.

Summary

The central takeaway of this lecture is that rewards cannot be taken for granted. In real-world RL applications, specifying the reward function is often the hardest part of the problem. This lecture covered three approaches to reward learning, each with distinct trade-offs:

Learning from Goals/Demos Learning from Preferences
Strengths Practical framework for task specification; only needs outcome examples Pairwise preferences are easy to provide; does not require demonstrations; has been deployed at scale (RLHF)
Weaknesses Adversarial training can be unstable; requires examples of desired behavior or outcomes May require supervision in the loop of RL; usually requires more total human time

The key ideas to take away are:

  1. Task specification is hard, and naive approaches (static classifiers, proxy rewards) can be exploited by RL agents.
  2. Goal classifiers with adversarial training provide a practical framework when examples of desired outcomes are available.
  3. The Bradley-Terry model provides a principled framework for learning rewards from pairwise preferences, leading to the widely-used RLHF pipeline.
  4. RLHF has been deployed at scale for LLM alignment, and RLAIF extends this by replacing human annotators with AI evaluators.
  5. Reward hacking remains a central challenge: any learned reward model can be exploited by a sufficiently powerful optimizer.