Recap: Policy Gradients
In the previous lecture we derived the REINFORCE estimator for the policy gradient. The core identity is
$$\nabla_\theta J(\theta) = \E_{\tau \sim \pi_\theta}\!\left[\sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \left(\sum_{t'=t}^{T-1} \gamma^{t'-t} r(s_{t'}, a_{t'})\right)\right].$$The quantity $\hat{Q}_t = \sum_{t'=t}^{T-1} \gamma^{t'-t} r(s_{t'}, a_{t'})$ is a single-sample Monte Carlo estimate of the action-value $\Qpi(s_t, a_t)$. While unbiased, this estimator can have extremely high variance because every reward along the trajectory contributes noise. Two classical strategies for variance reduction are:
- Causality ("rewards-to-go"): Drop rewards from time steps before $t$, since $\nabla_\theta \log \pi_\theta(a_t \mid s_t)$ cannot influence past rewards.
- Baselines: Subtract a state-dependent baseline $b(s_t)$ from the return without introducing bias, because $\E_{a_t}\!\left[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, b(s_t)\right] = 0$.
This lecture develops both ideas to their logical conclusion: the actor-critic architecture, which learns an explicit critic (value function) and uses it to construct low-variance estimates of the policy gradient for the actor (policy).
Value Functions and the Advantage
Before diving into actor-critic methods, let us revisit the three fundamental objects of policy evaluation.
The state-value function under policy $\pi$ is the expected discounted return starting from state $s$:
$$\Vpi(s) = \E_{\pi}\!\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \;\middle|\; s_0 = s\right].$$The action-value function additionally conditions on taking action $a$ in the first step:
$$\Qpi(s, a) = \E_{\pi}\!\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \;\middle|\; s_0 = s,\, a_0 = a\right].$$The advantage function measures how much better action $a$ is compared to the average action under $\pi$:
$$A^\pi(s, a) = \Qpi(s, a) - \Vpi(s).$$By construction, $\E_{a \sim \pi(\cdot|s)}[A^\pi(s,a)] = 0$ for all $s$.
The advantage function is the ideal quantity to use in the policy gradient because it has zero mean under the current policy. Intuitively, $A^\pi(s,a) > 0$ means action $a$ is better than average in state $s$, and the gradient update should increase its probability; $A^\pi(s,a) < 0$ means it is worse, and its probability should decrease.
The Value Function as a Baseline
Recall that for any baseline $b(s_t)$, the policy gradient can be written as
$$\nabla_\theta J(\theta) = \E_{\pi_\theta}\!\left[\sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\bigl(\hat{Q}_t - b(s_t)\bigr)\right].$$While any $b(s_t)$ leaves the gradient unbiased, the variance of the estimator depends on the choice of baseline. A natural and near-optimal choice is $b(s_t) = \Vpi(s_t)$, because then the gradient becomes
$$\nabla_\theta J(\theta) \approx \frac{1}{N}\sum_{i=1}^{N}\sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t^i \mid s_t^i)\, A^\pi(s_t^i, a_t^i).$$Using $\Vpi(s)$ as the baseline converts the policy gradient into an advantage-weighted gradient. The advantage has zero mean by construction, which dramatically reduces variance compared to using raw returns. The better the value function estimate, the lower the variance of the gradient.
Of course, we do not know $\Vpi$ exactly—we must learn it. This is the role of the critic in actor-critic methods.
Estimating the Advantage
Given a learned value function approximation $V_\phi(s) \approx \Vpi(s)$, there are several ways to estimate the advantage $A^\pi(s_t, a_t)$. Each trades off bias and variance differently.
Monte Carlo Estimate
The simplest approach uses the single-sample return as an estimate of $\Qpi$:
$$\hat{A}_t^{\text{MC}} = \left(\sum_{t'=t}^{T-1} \gamma^{t'-t} r(s_{t'}, a_{t'})\right) - V_\phi(s_t).$$This is unbiased (the return is an unbiased estimate of $\Qpi$) but has high variance because it sums many stochastic reward terms.
One-Step TD (Temporal Difference) Estimate
At the other extreme, we can use a single Bellman step:
$$\hat{A}_t^{\text{TD}} = r(s_t, a_t) + \gamma V_\phi(s_{t+1}) - V_\phi(s_t).$$This is the TD residual, often written $\delta_t$. It has low variance (only one reward term), but introduces bias if $V_\phi$ is not a perfect approximation of $\Vpi$.
$n$-Step Estimate
A natural interpolation uses $n$ actual rewards before bootstrapping:
$$\hat{A}_t^{(n)} = \left(\sum_{l=0}^{n-1} \gamma^l r(s_{t+l}, a_{t+l})\right) + \gamma^n V_\phi(s_{t+n}) - V_\phi(s_t).$$Larger $n$ reduces bias (more real rewards, less reliance on a potentially inaccurate $V_\phi$) but increases variance (more stochastic terms).
Consider a 100-step episode. With $n=1$ (TD), the advantage estimate depends on a single reward and two value-function evaluations—low variance but potentially biased. With $n=100$ (Monte Carlo), the estimate sums all 100 rewards—unbiased but noisy. With $n=5$, we get a compromise: five real rewards plus a bootstrap, moderately low variance with moderate bias. In practice, $n$ between 5 and 20 often works well, but the optimal choice is task-dependent.
Generalized Advantage Estimation (GAE)
Rather than committing to a single $n$, Generalized Advantage Estimation (Schulman et al., 2016) takes an exponentially weighted average over all $n$-step estimates, producing a smooth interpolation controlled by a single parameter $\lambda \in [0, 1]$.
Define the TD residual at time $t$ as
$$\delta_t = r(s_t, a_t) + \gamma V_\phi(s_{t+1}) - V_\phi(s_t).$$Then GAE is defined as
$$\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}.$$Derivation: GAE as a weighted average of $n$-step returns
The $n$-step advantage can be written in terms of TD residuals:
$$\hat{A}_t^{(n)} = \sum_{l=0}^{n-1} \gamma^l \delta_{t+l}.$$To see this, expand the definition:
$$\hat{A}_t^{(n)} = -V_\phi(s_t) + r_t + \gamma r_{t+1} + \cdots + \gamma^{n-1} r_{t+n-1} + \gamma^n V_\phi(s_{t+n}).$$Now take the GAE definition as the weighted sum:
$$\hat{A}_t^{\text{GAE}} = (1-\lambda)\bigl(\hat{A}_t^{(1)} + \lambda\, \hat{A}_t^{(2)} + \lambda^2\, \hat{A}_t^{(3)} + \cdots\bigr).$$Substituting the TD-residual form and collecting terms, each $\delta_{t+l}$ appears in all $n$-step estimates with $n > l$. The coefficient of $\delta_{t+l}$ is
$$(1-\lambda)\sum_{n=l+1}^{\infty} \lambda^{n-1} \gamma^l = (1-\lambda) \cdot \gamma^l \cdot \frac{\lambda^l}{1-\lambda} = (\gamma\lambda)^l.$$Therefore $\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^l \delta_{t+l}$, as claimed.
GAE smoothly interpolates between two extremes:
- $\lambda = 0$: $\hat{A}_t = \delta_t$ (one-step TD). Low variance, high bias.
- $\lambda = 1$: $\hat{A}_t = \sum_{l=0}^{\infty} \gamma^l \delta_{t+l} = \hat{A}_t^{\text{MC}}$ (Monte Carlo). High variance, zero bias (given that $V_\phi$ is used only as a baseline).
In practice, $\lambda \in [0.9, 0.99]$ is common, striking a good balance between bias and variance.
Fitting the Value Function (The Critic)
The critic approximates $\Vpi(s)$ with a parameterised function $V_\phi(s)$, typically a neural network. Training the critic is a supervised regression problem: we minimise
$$\mathcal{L}(\phi) = \frac{1}{2}\sum_{t} \bigl\|V_\phi(s_t) - \hat{V}_t\bigr\|^2,$$where $\hat{V}_t$ is a target value. Two common choices for the target:
Monte Carlo Target
Use the actual discounted return from the trajectory:
$$\hat{V}_t^{\text{MC}} = \sum_{t'=t}^{T-1} \gamma^{t'-t} r(s_{t'}, a_{t'}).$$This is an unbiased target but can be noisy. It requires waiting until the end of the episode to compute.
Bootstrapped (TD) Target
Use the one-step Bellman target:
$$\hat{V}_t^{\text{TD}} = r(s_t, a_t) + \gamma V_\phi(s_{t+1}).$$This is biased (it depends on the current, imperfect $V_\phi$) but has lower variance and can be computed online without waiting for the episode to end. In practice, the bootstrapped target is overwhelmingly the more popular choice.
Combining function approximation, bootstrapping, and off-policy learning can lead to divergence—a phenomenon known as the "deadly triad" in RL. On-policy actor-critic uses the first two but avoids the third, making it relatively stable. When we move to off-policy methods in Lecture 5, managing this instability becomes a central challenge.
The Batch Actor-Critic Algorithm
Putting the actor and critic together, we obtain the batch actor-critic algorithm. In each iteration, we collect a batch of trajectories, use them to update both the critic and the actor, then discard the data and repeat. This is an on-policy algorithm: the data must come from the current policy $\pi_\theta$.
- Initialise actor parameters $\theta$ and critic parameters $\phi$.
- Repeat:
- Collect a batch of trajectories $\{(s_t, a_t, r_t, s_{t+1})\}$ by running $\pi_\theta$ in the environment.
- Compute advantage estimates $\hat{A}_t$ using the critic $V_\phi$ (e.g., via GAE).
- Update critic: Minimise $\mathcal{L}(\phi) = \frac{1}{2}\sum_t \|V_\phi(s_t) - \hat{V}_t\|^2$ by gradient descent on $\phi$.
- Update actor: Take a gradient step on $\theta$ using $$\nabla_\theta J(\theta) \approx \frac{1}{|\mathcal{B}|}\sum_{t \in \mathcal{B}} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \hat{A}_t.$$
A key practical detail is that the critic update and the advantage computation are interleaved. Some implementations fit the critic before computing advantages (so advantages use an up-to-date $V_\phi$); others compute advantages first and then update the critic. Both work in practice, though the former is slightly more common.
A2C and A3C
The Advantage Actor-Critic (A2C) and its asynchronous variant A3C (Mnih et al., 2016) are two of the most well-known instantiations of the actor-critic framework.
Asynchronous Advantage Actor-Critic (A3C)
A3C runs multiple parallel workers, each with its own copy of the environment. Each worker independently collects a short trajectory segment (e.g., $n = 5$ or $20$ steps), computes the policy gradient and critic loss, and asynchronously updates a shared set of global parameters. The key idea is that asynchronous updates from diverse workers provide natural exploration and decorrelate the gradient updates, functioning as an alternative to experience replay.
- Sync local parameters $\theta', \phi'$ with global parameters $\theta, \phi$.
- Collect $n$-step trajectory segment using $\pi_{\theta'}$.
- Compute $n$-step returns and advantage estimates $\hat{A}_t$.
- Compute gradients $\nabla_{\theta'} J$ and $\nabla_{\phi'} \mathcal{L}$.
- Asynchronously update global $\theta, \phi$ using the computed gradients.
- Go to step 1.
Synchronous Advantage Actor-Critic (A2C)
A2C is the synchronous version: all workers collect trajectories in parallel, but gradients are aggregated before a single, synchronised update. Empirically, A2C matches or exceeds A3C in performance while being simpler to implement and debug. It also makes more efficient use of GPU batching.
A3C's asynchronous updates can cause workers to compute gradients based on stale parameters, which introduces noise and can slow convergence. A2C avoids this by synchronising updates. With modern vectorised environments (e.g., batched simulation on a single GPU), A2C can collect data just as fast as A3C without the staleness problem. This is why A2C has largely supplanted A3C in practice.
Online Actor-Critic
The batch algorithm collects full trajectories or large batches before updating. An alternative is the online (single-step) actor-critic, which updates after every environment step:
- Observe state $s$, take action $a \sim \pi_\theta(\cdot \mid s)$, observe $r, s'$.
- Compute TD residual: $\delta = r + \gamma V_\phi(s') - V_\phi(s)$.
- Update critic: $\phi \leftarrow \phi - \alpha_\phi\, \delta\, \nabla_\phi V_\phi(s)$.
- Update actor: $\theta \leftarrow \theta + \alpha_\theta\, \delta\, \nabla_\theta \log \pi_\theta(a \mid s)$.
- Set $s \leftarrow s'$ and repeat.
The online actor-critic is appealingly simple: the TD error $\delta$ serves simultaneously as the advantage estimate $\hat{A}(s,a) \approx \delta$ and the signal for updating the critic. However, single-step updates tend to be very noisy and can be unstable with neural-network function approximators. In practice, the batch approach with GAE is more robust and is the standard choice for deep RL.
Discount Factors in Practice
The discount factor $\gamma$ plays a dual role in actor-critic methods. First, it defines the objective: we optimise the $\gamma$-discounted return. Second, it controls the effective horizon of the value function and advantage estimates.
In many practical implementations, the discount factor is applied only to the critic's bootstrapping and not explicitly used as a weighting in the policy gradient sum. The rationale is that for episodic tasks, we typically want to optimise the undiscounted return $\sum_t r_t$ but use $\gamma < 1$ as a variance reduction technique that limits how far into the future the critic must predict.
In locomotion tasks (e.g., MuJoCo Humanoid), a typical configuration is $\gamma = 0.99$ and $\lambda = 0.95$. The relatively high $\gamma$ ensures the agent plans over long horizons (crucial for walking), while $\lambda < 1$ keeps the advantage estimates stable. In short-horizon tasks (e.g., CartPole), smaller values like $\gamma = 0.95$ can work because the relevant planning horizon is shorter.
Architecture Choices
A key design decision in actor-critic methods is whether the actor and critic share parameters or use separate networks.
Shared Network
A common architecture uses a shared feature extractor (e.g., a convolutional network for image inputs) with two separate heads: one outputting the policy $\pi_\theta(a \mid s)$ and one outputting $V_\phi(s)$. This is parameter-efficient and can accelerate learning when useful representations are common to both tasks. However, the actor and critic losses can sometimes conflict, requiring careful loss weighting.
Separate Networks
Using entirely separate networks for the actor and critic avoids gradient interference. This is often preferred when the observation space is low-dimensional (e.g., joint angles) and the computational cost of separate networks is negligible. Most modern implementations (e.g., PPO in practice) lean toward separate networks or very lightly shared architectures.
Nearly all modern deep RL algorithms are actor-critic methods at heart. PPO, SAC, TD3, and even many offline RL algorithms use a learned critic to guide policy updates. The ideas in this lecture—advantage estimation, GAE, and the interplay of actor and critic updates—recur throughout the rest of the course.
Summary
In this lecture we built the actor-critic framework from first principles. The key ideas are:
- The advantage function $A^\pi(s,a) = \Qpi(s,a) - \Vpi(s)$ is the ideal quantity for the policy gradient, since it has zero mean and directly indicates which actions are better than average.
- A learned critic $V_\phi(s) \approx \Vpi(s)$ provides a baseline that dramatically reduces variance.
- GAE smoothly interpolates between low-variance/high-bias (TD) and high-variance/low-bias (MC) advantage estimates via the parameter $\lambda$.
- The batch actor-critic algorithm alternates between collecting on-policy data, fitting the critic, and updating the actor with advantage-weighted policy gradients.
- A2C/A3C scale the algorithm across parallel workers, with synchronous updates (A2C) generally preferred in modern practice.
The main limitation of on-policy actor-critic is sample efficiency: every gradient update requires fresh data from the current policy. In the next lecture, we will explore off-policy actor-critic methods that reuse past experience, dramatically improving data efficiency.