Lecture 7

Offline RL

Learning from fixed datasets without further environment interaction: distributional shift, conservative methods, and practical algorithms.

Offline RL CQL IQL Distributional Shift
Original PDF slides

Why Offline RL?

Throughout this course, we have studied reinforcement learning algorithms that alternate between collecting data in the environment and improving a policy. Whether on-policy (collecting fresh data each iteration) or off-policy (reusing a replay buffer), these methods assume the agent can continuously interact with the environment. But what if we cannot?

In many real-world settings, online data collection is expensive, dangerous, or simply infeasible. A hospital cannot randomly assign treatments to patients just to explore. A self-driving car company cannot deploy untested policies on public roads. A robotics lab may have terabytes of data from previous experiments but no budget for new ones. These scenarios motivate offline reinforcement learning—also called batch RL—where we must learn the best possible policy from a fixed, pre-collected dataset, with no further environment interaction.

Definition
Offline Reinforcement Learning. Given a static dataset $\mathcal{D} = \{(s, a, s', r)\}$ collected by some unknown behavior policy $\piB$, learn a policy $\pi_\theta$ that maximizes the expected cumulative reward: $$\max_\theta \; \E_{p_{\theta}(\tau)}\!\left[\sum_t r(s_t, a_t)\right]$$ The key constraint is that the expectation under the learned policy $\pi_\theta$ must be optimized using only data from $\piB$.

Online vs. Offline

The distinction is straightforward but consequential:

Offline RL is especially valuable when we can leverage datasets collected by people, hand-designed controllers, or previous RL experiments. It also enables reuse of data across projects and institutions, and it avoids the safety risks of deploying partially-trained policies. A common practical workflow is offline pre-training followed by online fine-tuning, combining the data efficiency of offline methods with the performance of online adaptation.

Where Does the Data Come From?

The offline dataset $\mathcal{D}$ can originate from many sources:

Key Insight
The behavior policy $\piB$ is typically unknown and may be a mixture of many different policies with varying quality levels. This heterogeneity is both a challenge (the data distribution is complex) and an opportunity (the dataset may contain some excellent trajectories alongside mediocre ones).

The Distributional Shift Problem

A natural first attempt at offline RL would be to simply run an off-policy algorithm—such as SAC—on the static dataset, treating it as a fixed replay buffer. After all, off-policy methods are designed to learn from data collected by a different policy. Can we just do that?

Unfortunately, the answer is no—at least not without modification. The core issue is distributional shift between the behavior policy $\piB$ and the learned policy $\pi_\theta$.

How Overestimation Arises

Recall the standard off-policy critic objective (e.g., from SAC):

$$\min_\phi \sum_{(s,a,s') \sim \mathcal{D}} \left\| \hat{Q}^{\pi_\theta}_\phi(s, a) - \left(r(s,a) + \gamma \E_{a' \sim \pi_\theta(\cdot|s')}[\hat{Q}^{\pi_\theta}_\phi(s', a')]\right) \right\|^2$$

The critical problem lies in the target computation: we must evaluate $\hat{Q}(s', a')$ where $a' \sim \pi_\theta(\cdot|s')$. But $\pi_\theta$ is being optimized to select actions that maximize $Q$, so it will choose actions where $Q$ is large. If those actions are out-of-distribution (OOD)—meaning they were rarely or never taken by $\piB$ at state $s'$—then the $Q$-function has never been trained on them and its output is essentially random.

This creates a vicious cycle:

  1. The randomly initialized $Q$-function has arbitrary values on OOD actions.
  2. The policy selects actions where $Q$ happens to be high (likely OOD actions, since the $Q$ landscape is noisy outside the data support).
  3. The Bellman backup propagates these overestimates to in-distribution state-action pairs.
  4. After the policy update, $Q$-values become substantially overestimated everywhere.
Example
The OOD Action Problem. Imagine a randomly initialized $Q$-function for some state $s'$. It produces an arbitrary, wavy curve over the action space. The dataset only contains actions in a narrow region (the "data support"). Outside this region, $Q$ may exhibit spurious peaks. The policy, seeking to maximize $Q$, will select these spurious peaks. Their inflated values then get backed up into the $Q$-estimates for other states, causing a cascading overestimation.

From another perspective, the learned policy deviates too much from the behavior policy. In online RL, this mismatch is self-correcting: the agent collects new data under $\pi_\theta$, and the $Q$-function is trained on that data. In offline RL, no such correction is possible. Mitigating this overestimation is the core goal of offline RL methods.

Offline RL versus Imitation Learning

Before diving into solutions, it is worth asking: why not just do imitation learning on the offline dataset? If we have demonstrations, we could simply clone the behavior policy via supervised learning. The answer lies in a fundamental limitation of imitation learning: it cannot outperform the demonstrator.

Offline data may not be optimal. It might contain a mixture of good and bad trajectories collected by different policies or by the same policy at different stages of learning. Offline RL can leverage reward information to do something imitation learning cannot: stitch together good segments from different trajectories to synthesize a policy better than any individual trajectory in the dataset.

Example
Trajectory Stitching. Consider a dataset with two trajectories: one that navigates well from $s_1$ to $s_3$, and another that navigates well from $s_7$ to $s_9$ (the goal). Neither trajectory alone reaches the goal from $s_1$. But by using reward information and dynamic programming, an offline RL method can learn a policy that goes from $s_1$ all the way to $s_9$, combining the best segments. Imitation learning, which only copies the demonstrator's actions, cannot discover this composite strategy.

This stitching ability is the key advantage of offline RL over imitation learning. It is also what makes the problem harder: stitching requires evaluating actions that no single trajectory in the dataset took, which brings us back to the distributional shift challenge.

Filtered and Weighted Imitation Learning

The simplest approach to using reward information in the offline setting is to stay close to imitation learning but incorporate reward signals. This gives us two simple baselines that are often surprisingly effective.

Filtered Behavior Cloning

The idea is straightforward: if we have reward labels, only imitate the good trajectories.

Filtered Behavior Cloning
  1. Rank trajectories by return: $r(\tau) = \sum_{(s_t, a_t) \in \tau} r(s_t, a_t)$
  2. Filter the dataset to include only the top $k\%$: $\tilde{D} = \{\tau \mid r(\tau) > \eta\}$
  3. Imitate the filtered dataset: $\max_\theta \sum_{(s,a) \in \tilde{D}} \log \pi_\theta(a \mid s)$

This is a very primitive approach to using reward information. It discards potentially useful transitions from mediocre trajectories and cannot perform trajectory stitching. However, it is simple and makes a good baseline to test against.

Advantage-Weighted Regression (AWR)

Rather than binary filtering, we can weight each transition by how good the action was. This leads to a more principled approach based on the advantage function.

Recall that the advantage function $A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)$ measures how much better action $a_t$ is compared to the average action under policy $\pi$. The advantage-weighted imitation objective is:

$$\theta \leftarrow \arg\max_\theta \; \E_{s, a \sim D}\!\left[\log \pi_\theta(a \mid s) \exp(A(s, a))\right]$$

This is simply standard imitation learning (the $\log \pi_\theta$ term) reweighted by the exponentiated advantage (the $\exp(A)$ term). Transitions with high advantage receive more weight; transitions with low advantage are effectively downweighted.

Theorem
Connection to KL-Constrained RL. The advantage-weighted regression objective can be shown to approximate the solution to the KL-constrained policy improvement problem: $$\pi_{\text{new}} = \arg\max_\pi \; \E_{a \sim \pi(\cdot|s)}[Q(s, a)] \quad \text{s.t.} \quad \Dkl(\pi \| \piB) < \epsilon$$ This means AWR implicitly keeps the learned policy close to the behavior policy while improving it.

The key question for AWR is: how do we estimate the advantage function? The simplest approach uses Monte Carlo estimation:

Advantage-Weighted Regression (AWR)
  1. Fit value function via Monte Carlo regression: $\min_\phi \sum_{s_t \sim \mathcal{D}} \left\| \hat{V}^{\piB}_\phi(s_t) - \sum_{t'=t}^{T} r(s_{t'}, a_{t'}) \right\|^2$
  2. Train policy with advantage weights: $\max_\theta \; \E_{(s_t, a_t) \sim \mathcal{D}} \left[\log \pi_\theta(a_t | s_t) \exp\!\left(\frac{1}{\alpha}\left(\sum_{t'=t}^{T} r(s_{t'}, a_{t'}) - \hat{V}^{\piB}_\phi(s_t)\right)\right)\right]$

The temperature $\alpha$ is a hyperparameter controlling how sharply the weights distinguish good from bad actions.

AWR has important advantages: it is simple, and it avoids querying $Q$-values on OOD actions entirely. However, Monte Carlo estimation is noisy, and the advantage $\hat{A}^{\piB}$ is computed for the behavior policy rather than the learned policy $\pi_\theta$, which limits how much improvement is possible.

Implicit Q-Learning (IQL)

Can we do better than AWR while still avoiding the OOD action problem? The key insight of Implicit Q-Learning (Kostrikov, Nair, and Levine, ICLR 2022) is to estimate Q-values using TD updates but to never query the $Q$-function on actions outside the dataset.

From AWAC to IQL

A first improvement over AWR is to estimate advantages using TD learning instead of Monte Carlo. The Advantage-Weighted Actor-Critic (AWAC) approach estimates the $Q$-function with:

$$\hat{Q}^{\piB} \leftarrow \arg\min_Q \; \E_{(s,a,s',a') \sim D}\!\left[\left(Q(s, a) - \left(r + \gamma Q(s', a')\right)\right)^2\right]$$

Here, the next action $a'$ is sampled from the dataset (not from $\pi_\theta$), so we never query $Q$ on OOD actions. This gives a $Q$-function estimate for $\piB$, not for $\pi_\theta$. But can we estimate $Q$ for a policy better than $\piB$?

Expectile Regression

The key idea in IQL is to replace the standard mean squared error loss with an asymmetric loss called the expectile regression loss. Instead of fitting the mean of a random variable (which standard $\ell_2$ loss does), expectile regression can fit a higher or lower "expectile"—analogous to a quantile but based on squared rather than absolute errors.

Definition
Expectile Regression Loss. For a parameter $\lambda \in (0, 1)$, the expectile loss is: $$\ell_2^\lambda(x) = \begin{cases} (1 - \lambda) x^2 & \text{if } x < 0 \\ \lambda x^2 & \text{otherwise} \end{cases}$$ When $\lambda = 0.5$, this reduces to the standard squared error. When $\lambda > 0.5$, the loss penalizes underestimation more heavily, pushing the fit toward higher values of the distribution. When $\lambda < 0.5$, it favors lower values.

Think of it this way: if we use $\ell_2$ loss to regress $V(s)$ onto the distribution of $Q(s, a)$ values for $a \sim \piB(\cdot|s)$, we recover the mean—i.e., $V^{\piB}(s)$. But if we use the asymmetric loss $\ell_2^\lambda$ with $\lambda > 0.5$, we recover a value closer to the maximum of the distribution—approximating the value function of the best policy within the data support.

The IQL Algorithm

IQL uses expectile regression to fit a value function for a policy that is implicitly better than the behavior policy, without ever querying $Q$ on OOD actions.

Implicit Q-Learning (IQL)

Repeat:

  1. Fit $V$ with expectile loss: $\hat{V}(s) \leftarrow \arg\min_V \; \E_{(s,a) \sim D}\!\left[\ell_2^\lambda\!\left(V(s) - \hat{Q}(s, a)\right)\right]$    using $\lambda < 0.5$
  2. Update $Q$ with standard MSE loss: $\hat{Q}(s, a) \leftarrow \arg\min_Q \; \E_{(s,a,s') \sim D}\!\left[\left(Q(s, a) - \left(r + \gamma \hat{V}(s')\right)\right)^2\right]$
  3. Extract policy with AWR: $\hat{\pi} \leftarrow \arg\max_\pi \; \E_{s,a \sim D}\!\left[\log \pi(a \mid s) \exp\!\left(\frac{1}{\alpha}\left(\hat{Q}(s, a) - \hat{V}(s)\right)\right)\right]$

The crucial point is that the $Q$-function is updated using $\hat{V}(s')$ instead of $\E_{a' \sim \pi}[\hat{Q}(s', a')]$. Since $V$ is a function of states only, no action sampling is needed, and we completely avoid querying $Q$ on OOD actions. Policy improvement happens implicitly through the asymmetric loss, hence the name "Implicit Q-Learning."

Key Insight
IQL decouples the actor and critic training in a way that eliminates the OOD action problem entirely. The value function $V$ is trained to approximate the value of the best policy in the data support via expectile regression, the $Q$-function is trained using $V$ as the bootstrap target (avoiding action queries), and the policy is extracted via advantage-weighted regression on in-distribution actions only.

Conservative Q-Learning (CQL)

While IQL avoids the OOD problem by never querying out-of-distribution actions, Conservative Q-Learning (Kumar et al., NeurIPS 2020) takes a different approach: it directly penalizes overestimated $Q$-values, producing a learned $Q$-function that is a lower bound on the true value.

The CQL Intuition

The idea is simple: in addition to the standard critic loss, add a regularizer that pushes down $Q$-values for actions that are likely to be overestimated. Specifically, CQL adds a term that minimizes $Q$-values under some broad distribution $\mu$ over actions:

$$\hat{Q}^\pi = \arg\min_Q \max_\mu \; \underbrace{\E_{(s,a,s') \sim D}\!\left[\left(Q(s, a) - \left(r(s,a) + \gamma \E_{a' \sim \pi}[Q(s', a')]\right)\right)^2\right]}_{\text{standard critic update}} + \alpha \underbrace{\E_{s \sim D, a \sim \mu(\cdot|s)}[Q(s, a)]}_{\text{push down on large } Q\text{-values}}$$
Theorem
CQL Lower Bound (Version 1). For sufficiently large $\alpha$, the CQL objective produces a $Q$-function satisfying $\hat{Q}^\pi(s, a) \leq Q^\pi(s, a)$ for all $(s, a)$. That is, the learned $Q$-function is a pointwise lower bound on the true $Q$-function.

Pushing Up In-Distribution Q-Values

The basic version above is too conservative: it pushes down $Q$-values everywhere, including for in-distribution actions where the estimates may be accurate. CQL addresses this by adding a counterbalancing term that pushes up $Q$-values for state-action pairs actually in the dataset:

$$\hat{Q}^\pi = \arg\min_Q \max_\mu \; \E_{(s,a,s') \sim D}\!\left[\left(Q(s, a) - \left(r + \gamma \E_\pi[Q(s', a')]\right)\right)^2\right] + \alpha \E_{s \sim D, a \sim \mu(\cdot|s)}[Q(s, a)] - \alpha \E_{(s,a) \sim D}[Q(s, a)]$$

The third term pushes up $Q$-values for $(s, a)$ pairs in the data. This means CQL pushes down on OOD actions and pushes up on in-distribution actions.

Theorem
CQL Lower Bound (Version 2). With the push-up term, it is no longer guaranteed that $\hat{Q}^\pi(s, a) \leq Q^\pi(s, a)$ for all individual $(s, a)$ pairs. However, the expected $Q$-value under the learned policy is still a lower bound: $$\E_{\pi(a|s)}[\hat{Q}^\pi(s, a)] \leq \E_{\pi(a|s)}[Q^\pi(s, a)] \quad \text{for all } s \in D$$ This is sufficient for safe policy improvement, since the policy uses expected $Q$-values for action selection.

The Full CQL Algorithm

Conservative Q-Learning (CQL)

Repeat:

  1. Update $\hat{Q}^\pi$ using the CQL loss $L_{\text{CQL}}$ on dataset $D$.
  2. Update policy $\pi$:
    • If actions are discrete: $\pi(a \mid s) = \begin{cases} 1 & \text{if } a = \arg\max_{\bar{a}} \hat{Q}^\pi(s, \bar{a}) \\ 0 & \text{otherwise} \end{cases}$
    • If actions are continuous: $\theta \leftarrow \theta + \eta \nabla_\theta \E_{s \sim D, a \sim \pi_\theta(\cdot|s)}[\hat{Q}^\pi(s, a)]$

Computing the CQL Objective in Practice

The CQL objective involves a maximization over the distribution $\mu$. With a maximum-entropy regularizer $R(\mu) = \E_{s \sim D}[\mathcal{H}(\mu(\cdot|s))]$, the optimal $\mu$ takes a simple closed form:

$$\mu(a \mid s) \propto \exp(Q(s, a))$$

Substituting this back, the penalty term becomes a log-sum-exp:

$$\E_{s \sim D, a \sim \mu(\cdot|s)}[Q(s, a)] = \log \sum_a \exp(Q(s, a))$$

This means we do not need to explicitly construct $\mu$—we simply compute the log-sum-exp of $Q$-values over the action space, which is straightforward for discrete actions and can be approximated by sampling for continuous actions.

Derivation of the log-sum-exp form

We want to solve $\max_\mu \E_{a \sim \mu(\cdot|s)}[Q(s, a)] + \mathcal{H}(\mu(\cdot|s))$. The entropy-regularized maximization has the well-known solution $\mu^*(a|s) \propto \exp(Q(s,a))$, i.e., $\mu^*(a|s) = \frac{\exp(Q(s,a))}{\sum_{a'} \exp(Q(s, a'))}$.

Substituting back:

$$\E_{a \sim \mu^*}[Q(s,a)] + \mathcal{H}(\mu^*) = \log \sum_a \exp(Q(s,a))$$

This is exactly the log-sum-exp (or "soft maximum") of the $Q$-values, which serves as a smooth upper envelope over actions. CQL penalizes this quantity, effectively pushing down $Q$-values where $Q$ is largest—precisely the OOD actions that may be overestimated.

A Practical Application: LinkedIn Notifications

Offline RL is not just a theoretical framework—it has seen real-world deployment. A compelling example comes from LinkedIn, which used CQL to optimize its notification-sending policy (Prabhakar, Yuan, Yang, Sun, and Muralidharan, 2022).

The challenge was multi-objective: maximize weekly active users (WAU) and click-through rate (CTR) while controlling the total volume of notifications. Online experimentation with RL would be risky—sending too many or poorly-timed notifications could drive users away. Instead, LinkedIn used offline RL on historical notification data.

Example
LinkedIn Notification Optimization. In online A/B testing, a standard DDQN policy (without conservatism) increased notification volume by 7.72% but decreased WAU by 0.69% and CTR by 7.79%. Adding CQL to the DDQN pipeline produced dramatically better results: sessions increased by 0.24%, WAU by 0.18%, and CTR by 2.26%, while reducing volume by 1.73%. The conservative penalty prevented the policy from overestimating the value of aggressive notification strategies.

Comparing Offline RL Algorithms

Given the variety of offline RL methods, how should a practitioner choose? The decision depends on the setting:

If You Only Want to Train Offline

If You Want Offline Pre-Training + Online Fine-Tuning

IQL appears to be the most performant choice for this workflow, as its implicit policy constraint makes it easier to transition to online data collection without catastrophic policy degradation.

Key Insight
Offline RL is still an active area of research. Recent extensions include IDQL (Hansen-Estruch et al., 2023), which combines IQL with diffusion policies for more expressive policy classes. The field is rapidly evolving, and new methods continue to push the boundary of what is possible from static datasets.

Summary

Offline reinforcement learning addresses the fundamental challenge of learning good policies from fixed datasets without further environment interaction. The key ideas covered in this lecture are:

  1. Why offline RL? Online data collection is expensive, risky, or infeasible in many real-world applications. Reusing offline data is both practical and efficient.
  2. The distributional shift challenge: Naively applying off-policy methods to static datasets leads to catastrophic overestimation of $Q$-values on out-of-distribution actions.
  3. Offline RL vs. imitation learning: Unlike imitation, offline RL can leverage reward information and trajectory stitching to learn policies that outperform the behavior policy.
  4. Filtered and weighted imitation: Simple baselines that incorporate reward information into behavior cloning, including advantage-weighted regression (AWR).
  5. Implicit policy constraint methods (IQL): Avoid OOD queries entirely by using expectile regression to implicitly estimate the value of an improved policy.
  6. Conservative methods (CQL): Directly penalize potentially overestimated $Q$-values to produce a lower-bound estimate, enabling safe policy improvement.

The central lesson is that successful offline RL requires explicit mechanisms to prevent the policy from exploiting errors in the learned value function at out-of-distribution actions. Whether through implicit constraints (IQL) or explicit conservatism (CQL), all effective offline RL methods address this core challenge.