What Is Imitation Learning?

In the previous lecture we introduced the Markov Decision Process and the idea that an agent interacts with an environment by choosing actions according to a policy $\pi$. But where does that policy come from? Reinforcement learning derives it from a reward signal—a scalar feedback that the agent seeks to maximize. Imitation learning takes a fundamentally different approach: instead of defining a reward function, we provide the agent with demonstrations of desired behavior and ask it to mimic the demonstrator.

The appeal is immediate. Specifying a reward function that captures every nuance of a complex task (drive safely, smoothly, and efficiently) is notoriously difficult. But collecting examples of a skilled human performing the task is often straightforward—we simply record what they do. Imitation learning converts those recordings into a policy.

A canonical example is autonomous driving. A dataset of human drivers provides sensor readings (camera images, LIDAR scans) paired with steering commands (wheel angle, throttle, brake). The imitation learning agent trains on these state-action pairs and, at deployment time, maps new sensor observations to driving actions. No reward function is specified—the demonstrations implicitly define what good behavior looks like.

Notation Recap

We will use the notation established in Lecture 1 throughout:

• State $s_t$ — the state of the world at time $t$.
• Observation $o_t$ — what the agent observes at time $t$ (equal to $s_t$ in fully-observed settings).
• Action $a_t$ — the decision taken at time $t$.
• Trajectory $\tau = (s_1, a_1, s_2, a_2, \ldots, s_T, a_T)$ — a sequence of states and actions.
• Policy $\pi(a_t \mid s_t)$ or $\pi(a_t \mid o_{t-m:t})$ — the agent's behavior, mapping observations to a distribution over actions. This can be represented using a generative model.
• Reward function $r(s, a)$ — how good a state-action pair is (not used directly in imitation learning).

Behavioral Cloning: The Simplest Approach

The most direct form of imitation learning is behavioral cloning (BC): treat the problem as supervised learning. We have inputs (states) and labels (expert actions), so we can train a neural network to map from one to the other using standard regression or classification losses.

Version 0: Deterministic Policy via Regression

The simplest behavioral cloning approach learns a deterministic policy $\hat{a} = \pi_\theta(s)$ by minimizing the mean squared error between the expert's actions and the policy's predictions:

This is precisely standard supervised regression. The neural network takes a state $s$ as input and outputs a predicted action $\hat{a}$. Training proceeds via stochastic gradient descent on the squared error, exactly as one would train a regression model on any tabular dataset. After training, the policy is deployed: the agent observes a state, queries the network, and executes the predicted action.

The Multimodality Problem

What goes wrong with $\ell_2$ regression? Consider an autonomous driving scenario where human drivers encounter a situation that admits two reasonable actions—for example, an obstacle in the road that can be avoided by steering either left or right. If the dataset contains demonstrations of both strategies, the $\ell_2$ loss will learn the mean of the two actions: steering straight ahead, which is precisely the worst thing to do.

The fundamental issue is that $\ell_2$ regression outputs a single point prediction, which is the conditional mean $\E[a \mid s]$. When the true conditional distribution $p(a \mid s)$ is multimodal, this mean can lie in a region of low probability—an action that no expert would ever take. This problem is not merely theoretical; it arises constantly in real-world imitation learning, particularly when demonstrations are collected from multiple people with different preferences and habits.

Learning Expressive Policy Distributions

The multimodality problem reveals that we need our policy to represent a distribution over actions, not just a single point. The question becomes: how do we parameterize $\pi_\theta(a \mid s)$ as an expressive distribution using a neural network?

Simple Parametric Distributions

For discrete actions (e.g., button presses in a video game), a neural network can output a categorical distribution directly. The network takes the state $s$ as input and outputs a probability for each action: $p(\text{up}), p(\text{down}), p(\text{left}), \ldots$ This representation is maximally expressive—it can represent any distribution over the finite action set, including multimodal ones.

For continuous actions (e.g., steering angle, joint torques), a common simple choice is a Gaussian distribution. The neural network outputs a mean $\mu$ and standard deviation $\sigma$, defining $\pi_\theta(a \mid s) = \mathcal{N}(\mu(s), \sigma(s)^2)$. While this is a valid probability distribution, it is unimodal—it can only represent a single bump. For the driving example above, a single Gaussian cannot capture the bimodal "go left or go right" structure. We need something more expressive.

Generative Models as Policies

The key insight of modern imitation learning is to borrow tools from generative modeling—the same family of techniques used for image generation and language modeling—and apply them to policy learning. In generative modeling, we learn complex distributions:

• Image diffusion models learn $p(\text{image} \mid \text{text description})$.
• Autoregressive language models learn $p(\text{next word} \mid \text{words so far})$.

For imitation learning, our goal is analogous: learn $p(\text{action} \mid \text{observations})$. Three prominent families of generative models have been adapted for policy learning:

Mixture of Gaussians

A Gaussian Mixture Model (GMM) represents the action distribution as a weighted sum of $K$ Gaussian components:

The neural network takes $s$ as input and outputs the parameters of all components: means $\mu_1, \ldots, \mu_K$, standard deviations $\sigma_1, \ldots, \sigma_K$, and mixture weights $w_1, \ldots, w_K$ (with $\sum_k w_k = 1$). Each component captures a different mode of the expert's behavior. For the driving example, one component might represent "merge left" and another "stay straight."

Discretize + Autoregressive Prediction

For high-dimensional continuous action spaces (e.g., a robot arm with many joints), the action vector $a_t \in \R^d$ can be decomposed dimension by dimension. First, each continuous dimension is discretized into bins. Then an autoregressive model predicts each dimension conditioned on the previous ones:

Each factor is a categorical distribution over the discretized bins for that dimension. This approach can represent arbitrarily complex distributions over actions because the chain rule decomposition loses no information, and categorical distributions over bins are maximally expressive for each conditional. This is the same principle that makes autoregressive language models so powerful.

Diffusion Policies

A diffusion policy generates actions by iteratively denoising a sample of Gaussian noise. Starting from pure noise $a_t^{(N)} \sim \mathcal{N}(0, I)$, the model applies $N$ steps of learned denoising, each conditioned on the current state $s_t$:

where $\epsilon_\theta$ is a neural network trained to predict the noise. At each step, the network takes as input the current noisy action $a_t^{(n)}$, the state $s_t$, and the denoising step index $n$, and outputs a noise estimate. After all $N$ steps, the resulting $a_t^{(0)}$ is the sampled action. Diffusion models can represent highly complex, multimodal distributions and have become a dominant approach in modern robot imitation learning.

Behavioral Cloning with Expressive Policies

Armed with expressive policy distributions, we can upgrade behavioral cloning. Instead of minimizing $\ell_2$ loss, we maximize the log-likelihood of the expert's actions under the policy distribution:

This objective is standard maximum likelihood estimation, the same principle that underlies training language models and image generation models. The key difference from Version 0 is that the policy $\pi_\theta$ is now an expressive distribution that can represent multiple modes, rather than a point prediction that can only capture the mean.

Empirical Evidence: Expressivity Matters

Experimental results consistently demonstrate the advantage of expressive policies, particularly with multimodal demonstration data:

• Simulated transport task (Chi et al., Diffusion Policy): With single-human demonstrations, both diffusion policies and GMM policies achieve high success rates (around 90% and 85%, respectively). But with multi-human demonstrations—where different people use different strategies—diffusion policies maintain roughly 90% success while GMM drops to around 40%.
• Real shirt-hanging task (Aloha Unleashed): With multi-human data, diffusion policies achieve around 70% success compared to roughly 20% for an L1-loss baseline.

The pattern is clear: when demonstration data comes from a single consistent demonstrator, even simple unimodal policies can work well. But when data is multimodal—whether from multiple demonstrators or from a single demonstrator facing ambiguous situations—expressive generative models are essential.

Case Studies in Industry

The combination of imitation learning with expressive policies has been adopted across robotics and autonomous driving:

• Robotics: Physical Intelligence's $\pi_0$ (diffusion), NVIDIA's GR00T N1 (diffusion), Figure Helix (diffusion), and OpenVLA (discretize + autoregressive) all use imitation learning with expressive generative policies to train robot manipulation and locomotion skills.
• Autonomous driving: Waymo's EMMA and Wayve's LINGO-2 both use discretize + autoregressive prediction to learn driving policies from human demonstration data.

The advantages of offline behavioral cloning are significant: there is no need for data from the learned policy (which can be unsafe or expensive to collect) and no need to define a reward function. The primary disadvantage is that it may require a very large amount of demonstration data for reliable performance, due to a fundamental problem we address next.

Compounding Errors and Distribution Shift

Even with a perfectly expressive policy class, behavioral cloning faces a fundamental challenge that distinguishes it from standard supervised learning: compounding errors. Understanding this phenomenon is essential for appreciating both the limitations of behavioral cloning and the motivation for more sophisticated imitation learning algorithms.

Supervised Learning vs. Sequential Decision Making

In standard supervised learning, the inputs $x_1, x_2, x_3, \ldots$ are drawn i.i.d. from some fixed distribution $p(x)$, and the predicted labels $\hat{y}$ have no effect on what the next input will be. Even if the model makes an error on input $x_i$, the next input $x_{i+1}$ is drawn from the same distribution—errors do not propagate.

In sequential decision making, the situation is fundamentally different. The predicted action $\hat{a}_t$ directly influences the next state $s_{t+1}$ via the environment's dynamics. If the policy makes even a small error at time $t$, it may end up in a state $s_{t+1}$ that is slightly different from any state the expert visited. At this new state, the policy has less reliable training signal (it may be outside the training distribution), so it is more likely to make another error, leading to a state even further from the expert's experience. Errors cascade: each mistake pushes the agent further from the data distribution, where further mistakes become increasingly likely.

Theoretical Analysis of Error Compounding

To make the compounding error problem precise, suppose the learned policy $\pi_\theta$ makes errors with probability $\epsilon$ at any given time step—that is, $\pi_\theta$ takes a different action from the expert with probability at most $\epsilon$ on states in the expert's distribution. How does the total error grow over a trajectory of length $T$?

In standard supervised learning (i.i.d. setting), the expected number of mistakes over $T$ inputs is simply $\epsilon T$—linear in the horizon. But in the sequential setting, errors compound. At time $t$, the probability of being in a state the expert would not have visited grows with each previous error. A careful analysis (Ross and Bagnell, 2010) shows that the total expected cost of the learned policy can be bounded as:

where $J(\pi)$ denotes the expected total cost (negative reward) of policy $\pi$ over a horizon of length $T$. The key term is the quadratic dependence on $T$: errors grow as $T^2$, not $T$. For a long horizon—say, driving for 30 minutes at 10 decisions per second—this quadratic blowup can make behavioral cloning unacceptably fragile.

Practical Implications

The $O(\epsilon T^2)$ bound has two immediate practical consequences:

1. Collect massive amounts of data. If we can drive $\epsilon$ down to a very small value through extensive demonstrations and powerful function approximation, the quadratic blowup may remain manageable. This is the approach taken by large-scale behavioral cloning systems that train on millions or even billions of demonstration trajectories.
2. Collect corrective data. Instead of trying to make $\epsilon$ vanishingly small, we can collect data that specifically addresses the distribution shift problem. This is the idea behind DAgger, which we discuss next.

DAgger: Dataset Aggregation

The compounding error problem arises because the policy is trained on the expert's state distribution but deployed on its own. DAgger (Dataset Aggregation), introduced by Ross, Gordon, and Bagnell (2011), addresses this directly by iteratively collecting expert labels on the states the learned policy actually visits.

The crucial insight is step (b): the expert provides labels not for the states it would visit, but for the states the learned policy visits. Over successive iterations, the training distribution converges to the distribution of states the policy actually encounters, eliminating the covariate shift problem. Ross et al. (2011) proved that DAgger achieves a regret bound of $O(\epsilon T)$—linear in the horizon, matching the standard supervised learning setting and eliminating the $T^2$ blowup.

The advantages of DAgger are clear: it is a data-efficient way to learn from an expert, directly addressing the compounding error problem. However, it has a significant practical limitation: querying the expert for actions at states visited by the learned policy can be challenging. The agent is in control of execution, and the expert must provide labels for potentially unusual or dangerous states that the agent has wandered into. In safety-critical domains like driving, this is problematic—the human expert must somehow specify the correct action from an awkward or dangerous situation that they would never have entered themselves.

Human-Gated DAgger: A Practical Variant

A more practical variant of DAgger is Human-Gated DAgger (HG-DAgger), which reverses the control structure. Instead of letting the learned policy execute fully and then querying the expert after the fact, HG-DAgger lets a human supervisor watch the policy execute and intervene whenever they judge the policy is about to make a mistake.

HG-DAgger has a much more practical interface than standard DAgger. The human watches the agent act and takes over when needed—exactly the paradigm of a driving instructor supervising a student driver. The expert provides full demonstrations from the intervention point onward, which is a natural and comfortable way to provide corrections. The downside is that it can be difficult to catch mistakes quickly in some application domains, especially when errors compound rapidly or the system operates at high speed.

An interesting research question, noted on the slides, is whether one could automatically detect when intervention is needed—for instance, by training an uncertainty estimator that flags states where the policy is likely to fail, triggering expert intervention without requiring constant human supervision.

How to Collect Demonstrations

The quality and practicality of imitation learning depends critically on how demonstration data is gathered. The challenge varies significantly across domains.

Naturally Occurring Demonstrations

In some domains, people already routinely perform the task being automated, and their behavior can simply be recorded. Autonomous driving is the prime example: billions of miles of human driving data exist in the form of dashcam footage paired with vehicle telemetry (steering angle, speed, braking). Similarly, language models can be trained on the vast corpus of human-written text—every sentence ever written is a "demonstration" of language generation.

Robotics Demonstrations

Robotics presents a harder challenge because humans cannot directly "do" what a robot does—we have different bodies, different sensors, and different actuators. Three common approaches for collecting robot demonstrations are:

• Kinesthetic teaching: A human physically moves the robot's joints through the desired motion. This provides an easy, intuitive interface, but the human is visible in the scene (potentially confusing vision-based policies) and it can be physically tiring.
• Remote controllers (teleoperation): The human operates the robot through a game controller, joystick, or VR device. Interface ease varies widely; some setups have significant latency between the human's commands and the robot's execution, which degrades demonstration quality.
• Puppeteering: A matching "leader" robot is manipulated by the human, and the "follower" robot mirrors the motions. This provides an intuitive interface with low latency, but requires building two copies of the robot hardware.

The Embodiment Gap

Could we avoid specialized data collection entirely by having robots learn from videos of humans or animals? This is an attractive idea—internet video data is plentiful and covers an enormous range of behaviors. However, the embodiment gap presents a fundamental obstacle:

• Appearance difference: A robot arm looks nothing like a human arm, so mapping visual observations across embodiments is non-trivial.
• Capability difference: Robots and humans have different degrees of freedom, joint limits, strength profiles, and dynamics. A motion that is natural for a human may be impossible for a given robot.

While it is hard to directly imitate human or animal demonstrations across the embodiment gap, such data can potentially guide exploration in reinforcement learning. For example, Peng et al. (SFV, SIGGRAPH Asia 2018) used video of humans performing athletic movements to define reward functions for training simulated characters, achieving impressive backflips and other acrobatic skills.

Theoretical Perspective: When Does Imitation Learning Work?

Let us consolidate the theoretical picture of when imitation learning succeeds and when it fails.

Behavioral Cloning Guarantees

Behavioral cloning treats imitation learning as an i.i.d. supervised learning problem. If the policy achieves expected per-step loss $\epsilon$ on the expert's state distribution, the performance guarantee is:

This quadratic dependence on horizon $T$ means that even small per-step errors can accumulate to produce catastrophic failure over long episodes. To achieve a total error of at most $\delta$, we need per-step error $\epsilon \leq O(\delta / T^2)$, which may require an impractically large number of demonstrations.

DAgger Guarantees

DAgger, by training on the policy's own state distribution, achieves a much stronger guarantee:

The linear dependence on $T$ matches the i.i.d. supervised learning rate. To achieve total error at most $\delta$, we need only $\epsilon \leq O(\delta / T)$, which is dramatically more lenient than the $O(\delta / T^2)$ requirement of behavioral cloning. The cost is that DAgger requires interactive access to the expert during training.

Summary and Looking Ahead

This lecture covered the two main paradigms of imitation learning, along with the critical issues that arise in practice:

Part 1: Offline Behavioral Cloning. Train a policy to mimic expert demonstrations using supervised learning. The key design choice is the policy's distribution family: simple regression produces the conditional mean (problematic for multimodal data), while expressive generative models—mixtures of Gaussians, autoregressive models, diffusion models—can capture the full distribution of expert behavior. The algorithm is fully offline, requiring no interaction with the environment. Its main weakness is susceptibility to compounding errors ($O(\epsilon T^2)$ cost growth) due to distribution shift between expert states and policy states.

Part 2: Online Imitation with DAgger. Address compounding errors by iteratively collecting expert labels on states the learned policy visits. DAgger achieves linear error growth ($O(\epsilon T)$), matching the i.i.d. supervised learning rate. Human-Gated DAgger provides a more practical variant where a human supervisor intervenes when the policy errs. The cost is the requirement for interactive expert access during training.

The strengths and limitations of imitation learning can be summarized as:

• Strengths: No need to define a reward function; offline BC requires no online data (which can be unsafe or expensive); DAgger provides a data-efficient path to reliable performance.
• Limitations: May need impractically large amounts of data; does not provide a framework for the agent to improve on its own through practice; performance is bounded by the quality of the expert.

In the next lecture, we begin our study of reinforcement learning algorithms, starting with policy gradient methods—techniques for directly optimizing a policy using reward signals from the environment, without any expert demonstrations.