The On-Policy Bottleneck
The batch actor-critic algorithm from Lecture 4 is on-policy: after each gradient update to $\theta$, we must discard all previously collected data and re-sample from the updated policy $\pi_\theta$. This is enormously wasteful—in robotics, for instance, a single trajectory might take minutes to collect, yet it is used for exactly one gradient step before being thrown away.
Off-policy methods break this bottleneck by maintaining a replay buffer $\mathcal{D}$ of past transitions and reusing them across many updates. The central challenge is that transitions $(s, a, r, s')$ stored in $\mathcal{D}$ were collected under older policies $\pi_{\theta_{\text{old}}}$, not the current $\pi_\theta$. We need techniques to correct for this distribution mismatch.
On-policy methods like A2C/PPO typically require $10^6$–$10^8$ environment steps on continuous control benchmarks. Off-policy methods like SAC can solve the same tasks with $10^5$–$10^6$ steps—an order-of-magnitude improvement. This matters enormously when data collection is expensive (real robots, complex simulators, human feedback).
Importance Sampling for Policy Gradients
One principled approach to off-policy correction is importance sampling. If transitions were collected under a behaviour policy $\piB$, we can reweight them to estimate expectations under $\pi_\theta$:
$$\E_{a \sim \pi_\theta}\!\left[f(a)\right] = \E_{a \sim \piB}\!\left[\frac{\pi_\theta(a \mid s)}{\piB(a \mid s)}\, f(a)\right].$$Applied to the policy gradient, this gives
$$\nabla_\theta J(\theta) = \E_{(s,a) \sim \piB}\!\left[\frac{\pi_\theta(a \mid s)}{\piB(a \mid s)}\, \nabla_\theta \log \pi_\theta(a \mid s)\, \hat{A}(s, a)\right].$$The ratio $w(s,a) = \pi_\theta(a \mid s) / \piB(a \mid s)$ is the importance weight. While this estimator is unbiased, it has a critical flaw.
For a transition $(s, a)$ collected under behaviour policy $\piB$, the importance weight is $w(s,a) = \frac{\pi_\theta(a \mid s)}{\piB(a \mid s)}$. This corrects for the mismatch between the data-generating distribution and the target distribution.
The problem is variance. If $\pi_\theta$ and $\piB$ diverge significantly, the importance weights can become very large (or very small), causing the gradient estimate to explode. Over a full trajectory of length $T$, the product of per-step importance weights $\prod_{t=0}^{T-1} w_t$ grows exponentially in $T$, making trajectory-level importance sampling essentially unusable for long horizons.
Modern off-policy actor-critic methods sidestep the importance-sampling problem entirely for the critic by exploiting the fact that the Bellman equation $\Qpi(s,a) = r(s,a) + \gamma \E_{s'}[\Vpi(s')]$ holds for any transition $(s, a, r, s')$, regardless of which policy generated it. This allows us to train the critic directly from replay-buffer data. The actor update then uses the critic's estimates rather than importance-weighted returns.
Replay Buffers
A replay buffer (or experience replay memory) is a data structure that stores transitions $(s, a, r, s')$ as they are collected and samples mini-batches uniformly at random for training. First introduced for DQN (Lecture 6) and later adopted by actor-critic methods, replay buffers provide two key benefits:
- Sample reuse: Each transition can be used for many gradient updates, improving data efficiency.
- Decorrelation: Randomly sampling from a buffer breaks the temporal correlations in sequential data, stabilising gradient-based learning with neural networks.
Typical buffer sizes range from $10^5$ to $10^6$ transitions. When the buffer is full, the oldest transitions are overwritten (FIFO). The ratio of gradient updates to environment steps—the update-to-data (UTD) ratio—is a key hyperparameter. Higher UTD ratios improve sample efficiency but can cause overfitting to stale data.
Deep Deterministic Policy Gradient (DDPG)
DDPG (Lillicrap et al., 2016) is an off-policy actor-critic algorithm designed for continuous action spaces. Its key innovation is using a deterministic policy $\mu_\theta(s)$ instead of a stochastic one, which avoids the need for importance sampling entirely.
The Deterministic Policy Gradient
For a deterministic policy $a = \mu_\theta(s)$, the policy gradient simplifies dramatically. Instead of $\nabla_\theta \log \pi_\theta(a \mid s)$, we use the chain rule through the Q-function:
For a deterministic policy $\mu_\theta$, the policy gradient is
$$\nabla_\theta J(\theta) = \E_{s \sim \mathcal{D}}\!\left[\nabla_a Q_\phi(s, a)\big|_{a=\mu_\theta(s)} \cdot \nabla_\theta \mu_\theta(s)\right].$$This requires no importance weights because the expectation is only over states $s$, which can be sampled from any distribution covering the state space (e.g., the replay buffer).
Intuition: Why deterministic gradients avoid importance sampling
In the stochastic case, the policy gradient involves $\E_{a \sim \pi_\theta}[\cdot]$, so off-policy data requires correcting the action distribution. With a deterministic policy, the action is a function of the state: $a = \mu_\theta(s)$. The gradient $\nabla_\theta J$ only involves an expectation over states, not actions. Since we only need states to be representative (not actions to be correctly distributed), we can use any state distribution, including one from a replay buffer.
DDPG Algorithm
DDPG combines the deterministic policy gradient with several stabilising techniques:
- Initialise actor $\mu_\theta$, critic $Q_\phi$, and target networks $\mu_{\theta'} \leftarrow \mu_\theta$, $Q_{\phi'} \leftarrow Q_\phi$.
- Initialise replay buffer $\mathcal{D}$.
- For each environment step:
- Select action $a = \mu_\theta(s) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma^2)$ (exploration noise).
- Execute $a$, observe $r, s'$. Store $(s, a, r, s')$ in $\mathcal{D}$.
- Sample mini-batch $\{(s_j, a_j, r_j, s_j')\}$ from $\mathcal{D}$.
- Compute target: $y_j = r_j + \gamma\, Q_{\phi'}(s_j', \mu_{\theta'}(s_j'))$.
- Update critic: $\phi \leftarrow \phi - \alpha_\phi \nabla_\phi \frac{1}{|\mathcal{B}|}\sum_j (Q_\phi(s_j, a_j) - y_j)^2$.
- Update actor: $\theta \leftarrow \theta + \alpha_\theta \frac{1}{|\mathcal{B}|}\sum_j \nabla_a Q_\phi(s, a)|_{a=\mu_\theta(s_j)} \nabla_\theta \mu_\theta(s_j)$.
- Soft update targets: $\theta' \leftarrow \tau\theta + (1-\tau)\theta'$, $\phi' \leftarrow \tau\phi + (1-\tau)\phi'$.
The target networks $\mu_{\theta'}$ and $Q_{\phi'}$ are slowly-updated copies of the main networks (Polyak averaging with $\tau \approx 0.005$). They stabilise the TD targets, preventing the feedback loop where the critic chases a rapidly moving target. Exploration is achieved by adding Gaussian or Ornstein-Uhlenbeck noise to the deterministic action.
Twin Delayed DDPG (TD3)
DDPG often suffers from overestimation bias in the critic and brittle training dynamics. TD3 (Fujimoto et al., 2018) addresses these issues with three targeted modifications.
Trick 1: Clipped Double Q-Learning
TD3 maintains two independent Q-networks, $Q_{\phi_1}$ and $Q_{\phi_2}$, and uses the minimum of their target values to form the Bellman target:
$$y = r + \gamma \min_{i=1,2} Q_{\phi_i'}(s', \tilde{a}'),$$where $\tilde{a}'$ is the target action (see Trick 3). Taking the minimum counteracts the positive bias that arises because $\max$ operations over noisy Q-estimates systematically overestimate true values.
When we compute $\max_a Q_\phi(s', a)$ to form the TD target, noise in $Q_\phi$ causes us to preferentially select actions whose Q-values are overestimated. This bias compounds over many Bellman backups, leading to wildly inflated Q-values and poor policies. Taking the minimum of two independent estimates provides a pessimistic correction that largely eliminates this problem.
Trick 2: Delayed Policy Updates
TD3 updates the actor (and target networks) less frequently than the critic—typically once every $d = 2$ critic updates. The rationale is that the actor should only be updated when the critic is sufficiently accurate. Updating the actor with noisy Q-estimates wastes gradient steps and can destabilise training.
Trick 3: Target Policy Smoothing
When computing the target value, TD3 adds clipped noise to the target action:
$$\tilde{a}' = \mu_{\theta'}(s') + \text{clip}(\epsilon, -c, c), \quad \epsilon \sim \mathcal{N}(0, \sigma^2).$$This acts as a regulariser: by smoothing the Q-function over nearby actions, it prevents the policy from exploiting narrow peaks in the Q-estimate that may be artifacts of function approximation error.
- Use two critics $Q_{\phi_1}, Q_{\phi_2}$; compute targets with $\min(Q_{\phi_1'}, Q_{\phi_2'})$.
- Update actor and target networks every $d$ critic updates (typically $d = 2$).
- Add clipped Gaussian noise to target actions for smoothing.
Maximum Entropy Reinforcement Learning
Before introducing SAC, we need the maximum entropy RL framework (Ziebart, 2010; Haarnoja et al., 2018). Instead of maximising the standard return, we augment the objective with an entropy bonus:
$$J_{\text{MaxEnt}}(\pi) = \sum_{t=0}^{\infty} \gamma^t\, \E_{(s_t, a_t) \sim \pi}\!\left[r(s_t, a_t) + \alpha\, \mathcal{H}\!\bigl(\pi(\cdot \mid s_t)\bigr)\right],$$where $\mathcal{H}(\pi(\cdot \mid s)) = -\E_{a \sim \pi}[\log \pi(a \mid s)]$ is the entropy of the policy and $\alpha > 0$ is the temperature parameter controlling the trade-off between reward maximisation and entropy maximisation.
Under the maximum entropy objective, the Bellman equations become "soft":
$$\Qpi_{\text{soft}}(s, a) = r(s, a) + \gamma\, \E_{s'}\!\left[\Vpi_{\text{soft}}(s')\right],$$ $$\Vpi_{\text{soft}}(s) = \E_{a \sim \pi}\!\left[\Qpi_{\text{soft}}(s, a) - \alpha \log \pi(a \mid s)\right].$$The optimal policy under this framework is the Boltzmann policy: $\pi^*(a \mid s) \propto \exp\!\bigl(\frac{1}{\alpha} \Qstar_{\text{soft}}(s, a)\bigr)$.
Adding entropy to the objective has several practical benefits:
- Exploration: The policy is incentivised to maintain stochasticity, preventing premature convergence to suboptimal deterministic strategies.
- Robustness: Entropy-regularised policies are more robust to model misspecification and perturbations.
- Multi-modality: When multiple strategies achieve similar returns, the entropy bonus encourages the policy to cover all of them rather than arbitrarily committing to one.
- Smoother optimisation: The entropy term makes the objective more concave, improving the optimisation landscape.
Soft Actor-Critic (SAC)
SAC (Haarnoja et al., 2018) is the leading off-policy actor-critic algorithm for continuous control. It combines maximum entropy RL with the twin-Q-network idea from TD3.
Critic Update
SAC trains two Q-networks $Q_{\phi_1}$ and $Q_{\phi_2}$ to minimise the soft Bellman error. The target is
$$y = r + \gamma\!\left(\min_{i=1,2} Q_{\phi_i'}(s', a') - \alpha \log \pi_\theta(a' \mid s')\right), \quad a' \sim \pi_\theta(\cdot \mid s').$$Note that the next action $a'$ is sampled from the current policy $\pi_\theta$, not a target policy, and the entropy term $-\alpha \log \pi_\theta(a' \mid s')$ appears in the target to account for the maximum entropy objective.
The Reparameterization Trick
To update the actor, SAC uses the reparameterization trick rather than the log-probability policy gradient. The idea is to express the stochastic action as a deterministic function of the state and an independent noise variable:
$$a = f_\theta(\epsilon;\, s), \quad \epsilon \sim \mathcal{N}(0, I).$$Concretely, with a Gaussian policy, $f_\theta(\epsilon; s) = \mu_\theta(s) + \sigma_\theta(s) \odot \epsilon$, where $\mu_\theta$ and $\sigma_\theta$ are the mean and standard deviation output by the policy network. This allows us to backpropagate through the action into the policy parameters via the chain rule.
The SAC actor objective is
$$J_\pi(\theta) = \E_{s \sim \mathcal{D}}\!\left[\E_{\epsilon \sim \mathcal{N}}\!\left[\alpha \log \pi_\theta(f_\theta(\epsilon; s) \mid s) - \min_{i=1,2} Q_{\phi_i}(s, f_\theta(\epsilon; s))\right]\right].$$The gradient with respect to $\theta$ flows through both the log-probability term and the Q-function via the action $f_\theta(\epsilon; s)$, yielding a low-variance gradient estimate.
Why reparameterization gives lower variance than log-probability gradients
The standard policy gradient $\nabla_\theta \log \pi_\theta(a \mid s) \cdot Q(s,a)$ treats the action as a sample and uses the score function to "guess" how changing $\theta$ would change the probability of that sample. This is fundamentally a zero-order (black-box) approach to the action dependence.
The reparameterization trick, by contrast, allows the gradient to flow through the action: $\nabla_\theta Q(s, f_\theta(\epsilon; s))$ uses the first-order structure of $Q$ with respect to $a$ and of $f_\theta$ with respect to $\theta$. This is analogous to the difference between finite-difference and analytic gradients—the latter has far lower variance. Empirically, reparameterized gradients can be 10–100x lower variance than score-function gradients.
Squashed Gaussian Policy
Since many environments have bounded action spaces (e.g., torques in $[-1, 1]$), SAC applies a tanh squashing to the Gaussian output:
$$a = \tanh(\mu_\theta(s) + \sigma_\theta(s) \odot \epsilon).$$The log-probability under this transformation requires a change-of-variables correction:
$$\log \pi_\theta(a \mid s) = \log \mathcal{N}(u;\, \mu_\theta, \sigma_\theta) - \sum_{i=1}^{d} \log(1 - a_i^2),$$where $u = \text{atanh}(a)$ is the pre-squashing action and $d$ is the action dimension.
Automatic Temperature Tuning
The temperature $\alpha$ is crucial: too high and the policy is too random; too low and it collapses to a deterministic mode. SAC automates the choice of $\alpha$ by framing it as a constrained optimisation problem:
$$\min_\alpha\; \E_{a \sim \pi_\theta}\!\left[-\alpha \log \pi_\theta(a \mid s) - \alpha\, \bar{\mathcal{H}}\right],$$where $\bar{\mathcal{H}}$ is a target entropy (commonly set to $-\dim(\mathcal{A})$). This adjusts $\alpha$ so that the policy's entropy stays close to the target: if entropy is too low, $\alpha$ increases to encourage more exploration; if entropy is too high, $\alpha$ decreases.
- Initialise policy $\pi_\theta$, two critics $Q_{\phi_1}, Q_{\phi_2}$, target critics $Q_{\phi_1'}, Q_{\phi_2'}$, temperature $\alpha$.
- Initialise replay buffer $\mathcal{D}$.
- For each environment step:
- Sample $a \sim \pi_\theta(\cdot \mid s)$ and execute; store $(s, a, r, s')$ in $\mathcal{D}$.
- Sample mini-batch from $\mathcal{D}$.
- Critic update: Minimise $\frac{1}{|\mathcal{B}|}\sum(Q_{\phi_i}(s,a) - y)^2$ for $i=1,2$, where $y = r + \gamma(\min_i Q_{\phi_i'}(s', a') - \alpha \log \pi_\theta(a' \mid s'))$ and $a' \sim \pi_\theta(\cdot \mid s')$.
- Actor update: Minimise $\E_{\epsilon}\!\left[\alpha \log \pi_\theta(a_\theta \mid s) - \min_i Q_{\phi_i}(s, a_\theta)\right]$ where $a_\theta = f_\theta(\epsilon; s)$.
- Temperature update: $\alpha \leftarrow \alpha - \beta \nabla_\alpha \E[-\alpha \log \pi_\theta(a \mid s) - \alpha \bar{\mathcal{H}}]$.
- Target update: $\phi_i' \leftarrow \tau \phi_i + (1-\tau)\phi_i'$ for $i=1,2$.
Comparing Off-Policy Actor-Critic Algorithms
Let us summarise the key differences among DDPG, TD3, and SAC:
| Feature | DDPG | TD3 | SAC |
|---|---|---|---|
| Policy type | Deterministic | Deterministic | Stochastic |
| Number of Q-networks | 1 | 2 | 2 |
| Exploration | Action noise | Action noise | Entropy bonus |
| Actor gradient | DPG | DPG | Reparameterized |
| Entropy regularisation | No | No | Yes (auto-tuned $\alpha$) |
In practice, SAC is the most widely used off-policy actor-critic algorithm for continuous control. Its automatic entropy tuning removes a sensitive hyperparameter, and the stochastic policy provides built-in exploration. TD3 remains a strong alternative, particularly in settings where deterministic policies are preferred or the entropy bonus is undesirable.
Summary
This lecture addressed the fundamental limitation of on-policy methods—sample inefficiency—and introduced off-policy actor-critic algorithms that reuse data via replay buffers. The key ideas are:
- Importance sampling provides a principled correction for off-policy data but suffers from high variance, motivating alternative approaches.
- DDPG uses a deterministic policy gradient, sidestepping importance sampling entirely and enabling off-policy learning from a replay buffer.
- TD3 stabilises DDPG with three modifications: clipped double Q-learning, delayed policy updates, and target policy smoothing.
- Maximum entropy RL augments the reward with an entropy bonus $\alpha \mathcal{H}(\pi)$, encouraging exploration and improving robustness.
- SAC combines entropy regularisation with the reparameterization trick and twin Q-networks, achieving state-of-the-art sample efficiency on continuous control benchmarks.
All of the methods in this lecture learn both a policy (actor) and a Q-function (critic). In the next lecture, we will ask: can we dispense with the actor entirely and extract a policy directly from the Q-function? This leads to Q-learning—the foundation of value-based deep RL.