Lecture 6: Q-Learning

From Actor-Critic to Pure Value-Based Methods

In Lectures 4 and 5 we studied actor-critic algorithms that maintain both a policy $\pi_\theta$ (actor) and a value function $Q_\phi$ (critic). A natural question arises: do we even need the actor? If we have a good estimate of the optimal action-value function $\Qstar(s, a)$, the optimal policy is simply

$$\pi^*(a \mid s) = \begin{cases} 1 & \text{if } a = \arg\max_{a'} \Qstar(s, a'), \\ 0 & \text{otherwise.}\end{cases}$$

This observation is the foundation of Q-learning: learn $\Qstar$ directly, and extract the policy by taking the $\arg\max$. No separate policy network is needed. The approach is conceptually simple, extremely powerful for discrete action spaces, and led to one of the landmark results in deep RL—the DQN agent that learned to play Atari games from raw pixels.

Tabular Q-Learning

The classical Q-learning update (Watkins, 1989) maintains a table $Q(s, a)$ for every state-action pair and updates it using the Bellman optimality equation:

$$Q(s, a) \leftarrow Q(s, a) + \alpha\!\left[r + \gamma \max_{a'} Q(s', a') - Q(s, a)\right].$$

Definition: Bellman Optimality Equation

The optimal action-value function satisfies

$$\Qstar(s, a) = r(s, a) + \gamma\, \E_{s' \sim p(\cdot|s,a)}\!\left[\max_{a'} \Qstar(s', a')\right].$$

This is a fixed-point equation: $\Qstar$ is the unique function satisfying it. Q-learning can be viewed as stochastic fixed-point iteration toward this solution.

A remarkable property of Q-learning is that it is off-policy by nature: the update uses $\max_{a'} Q(s', a')$ regardless of which action the agent actually took. This means we can learn $\Qstar$ from data collected by any exploratory policy (e.g., $\epsilon$-greedy), as long as every state-action pair is visited infinitely often. In the tabular setting, Q-learning converges to $\Qstar$ under mild conditions.

Fitted Q-Iteration

For large or continuous state spaces, we cannot maintain a table. Instead, we approximate $\Qstar$ with a parameterised function $Q_\phi(s, a)$—typically a neural network. The fitted Q-iteration algorithm is the batch version of approximate Q-learning:

Fitted Q-Iteration

Collect a dataset $\mathcal{D} = \{(s_j, a_j, r_j, s_j')\}$ using some exploration policy.
Repeat until convergence:
1. Compute targets: $y_j = r_j + \gamma \max_{a'} Q_\phi(s_j', a')$ for all $(s_j, a_j, r_j, s_j') \in \mathcal{D}$.
2. Update $\phi$ by regression: $\phi \leftarrow \arg\min_\phi \sum_j \bigl(Q_\phi(s_j, a_j) - y_j\bigr)^2$.

Note the key subtlety: the targets $y_j$ depend on the current parameters $\phi$, making this a moving-target regression problem. Each time we update $\phi$, the targets shift. In the tabular case this converges, but with function approximation, convergence is not guaranteed—the combination of bootstrapping, function approximation, and off-policy data is notoriously unstable (the "deadly triad").

Deep Q-Networks (DQN)

The DQN algorithm (Mnih et al., 2013, 2015) demonstrated that deep Q-learning can work at scale by introducing two critical stabilisation techniques: experience replay and target networks. DQN learned to play 49 Atari games from raw pixel inputs, achieving human-level performance on many of them—a breakthrough that catalysed the modern deep RL era.

Experience Replay

DQN stores every transition $(s, a, r, s')$ in a large replay buffer $\mathcal{D}$ (typically holding the last $10^6$ transitions) and samples uniformly random mini-batches for training. Experience replay serves two purposes:

Data efficiency: Each transition is used for many gradient updates instead of being discarded after one.
Breaking correlations: Consecutive transitions in a trajectory are highly correlated. Training on sequential data causes the network to overfit to recent experience and forget earlier lessons. Random sampling from the buffer produces approximately i.i.d. mini-batches, which is what stochastic gradient descent assumes.

Key Insight: Replay as Regularisation

Experience replay can be understood as a form of implicit regularisation. By training on a diverse mix of old and new transitions, the Q-network is forced to maintain accurate predictions across the entire state space rather than just the states currently being visited. This dramatically reduces the catastrophic forgetting that plagues online neural-network training.

Target Networks

The second stabilisation technique is the target network. Instead of computing the TD target using the current Q-network $Q_\phi$, DQN maintains a separate target network $Q_{\phi^-}$ whose parameters are copied from $Q_\phi$ every $C$ steps (typically $C = 10{,}000$):

$$y_j = r_j + \gamma \max_{a'} Q_{\phi^-}(s_j', a').$$

Between copies, $\phi^-$ is frozen. This breaks the harmful feedback loop where the Q-network chases its own rapidly changing predictions. With a fixed target, each phase of training reduces to a standard supervised regression problem.

Definition: Polyak Averaging (Soft Target Updates)

An alternative to hard copying every $C$ steps is Polyak averaging, where the target network is updated after every step as a slow-moving exponential average:

$$\phi^- \leftarrow \tau\, \phi + (1 - \tau)\, \phi^-, \quad \tau \ll 1.$$

Typical values are $\tau \in [0.001, 0.01]$. This produces smoother target changes and is used by DDPG, TD3, and SAC.

The DQN Algorithm

Deep Q-Network (DQN)

Initialise Q-network $Q_\phi$ and target network $Q_{\phi^-} \leftarrow Q_\phi$.
Initialise replay buffer $\mathcal{D}$.
For each environment step $t = 1, 2, \ldots$:
1. With probability $\epsilon$ select random action $a_t$; otherwise $a_t = \arg\max_a Q_\phi(s_t, a)$.
2. Execute $a_t$, observe $r_t, s_{t+1}$. Store $(s_t, a_t, r_t, s_{t+1})$ in $\mathcal{D}$.
3. Sample mini-batch $\{(s_j, a_j, r_j, s_j')\}$ from $\mathcal{D}$.
4. Compute targets: $y_j = r_j + \gamma \max_{a'} Q_{\phi^-}(s_j', a')$.
5. Update: $\phi \leftarrow \phi - \alpha \nabla_\phi \frac{1}{|\mathcal{B}|}\sum_j (Q_\phi(s_j, a_j) - y_j)^2$.
6. Every $C$ steps: $\phi^- \leftarrow \phi$.

Example: DQN on Atari

The original DQN used a convolutional neural network that takes a stack of four $84 \times 84$ grayscale frames as input and outputs Q-values for each of the 18 possible joystick actions. The replay buffer held $10^6$ transitions. Training ran for 50 million frames (about 38 days of game time) with $\epsilon$ annealed from 1.0 to 0.1 over the first million frames. The same architecture and hyperparameters were used across all 49 games—no game-specific tuning.

The Overestimation Problem

Q-learning uses $\max_{a'} Q_\phi(s', a')$ to form the TD target. When $Q_\phi$ contains approximation errors (which it always does with function approximation), the $\max$ operator introduces a systematic positive bias.

Theorem: Overestimation Under Noise

Let $Q_\phi(s', a') = \Qstar(s', a') + \epsilon_{a'}$ where $\epsilon_{a'}$ are zero-mean noise terms. Then

$$\E\!\left[\max_{a'} Q_\phi(s', a')\right] \geq \max_{a'} \Qstar(s', a').$$

The inequality is strict whenever the noise terms are non-degenerate. The bias grows with the number of actions and the magnitude of the noise.

Proof sketch

By Jensen's inequality applied to the convex function $\max$:

$$\E[\max_{a'} Q_\phi(s', a')] = \E[\max_{a'}(\Qstar(s', a') + \epsilon_{a'})] \geq \max_{a'} \E[\Qstar(s', a') + \epsilon_{a'}] = \max_{a'} \Qstar(s', a'),$$

since each $\epsilon_{a'}$ has zero mean. The inequality is strict when the $\epsilon_{a'}$ are non-degenerate because $\max$ is strictly convex over distributions with non-trivial support. Intuitively, the $\max$ operation preferentially selects the action whose Q-value is most overestimated, systematically inflating the target.

This overestimation compounds through the Bellman backup: overestimated targets lead to overestimated Q-values, which produce even more overestimated targets. The result can be dramatic: Q-values may diverge to unreasonably large magnitudes, leading to poor policies that chase phantom high-value actions.

Double DQN

Double Q-learning (Van Hasselt, 2010) addresses overestimation by decoupling action selection from action evaluation. The key idea is to use the current network to select the best action, but the target network to evaluate it:

$$y_j^{\text{DDQN}} = r_j + \gamma\, Q_{\phi^-}\!\bigl(s_j',\, \arg\max_{a'} Q_\phi(s_j', a')\bigr).$$

Key Insight: Why Double DQN Works

Standard DQN uses the same network for both selecting and evaluating the best action: $\max_{a'} Q_{\phi^-}(s', a')$. If $Q_{\phi^-}$ overestimates the value of some action $a'$, that same overestimation directly enters the target. Double DQN breaks this coupling: even if $Q_\phi$ selects an overestimated action, $Q_{\phi^-}$ provides an independent evaluation that is unlikely to share the same bias. The result is a more accurate, less inflated target.

Double DQN (Van Hasselt et al., 2016) integrates this idea into the DQN framework with minimal changes—only the target computation differs. It requires no additional networks or computation, yet consistently reduces overestimation and improves performance across Atari games.

Further DQN Extensions

Multi-Step Returns

Just as we used $n$-step returns for advantage estimation in actor-critic methods, we can use them for Q-learning targets:

$$y_j^{(n)} = \sum_{l=0}^{n-1} \gamma^l r_{t+l} + \gamma^n \max_{a'} Q_{\phi^-}(s_{t+n}, a').$$

Multi-step targets propagate reward information faster (reducing the number of Bellman backups needed to propagate a distant reward) at the cost of introducing some off-policy bias (the intermediate actions may not be optimal). In practice, $n = 3$ to $5$ often works well, and the speed-up in learning can be substantial.

Dueling DQN

The dueling architecture (Wang et al., 2016) modifies the network structure to separately estimate the state-value $V(s)$ and the advantage $A(s, a)$:

$$Q_\phi(s, a) = V_\psi(s) + A_\xi(s, a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A_\xi(s, a').$$

The subtraction of the mean advantage ensures identifiability (otherwise $V$ and $A$ are only determined up to an additive constant). The intuition is that in many states, the choice of action matters little—the state value dominates. The dueling architecture can learn $V(s)$ efficiently from all transitions, even those where the action choice was irrelevant.

Example: When Dueling Helps

In the Atari game "Enduro" (a racing game), the agent spends most of its time driving straight. In these states, all actions (left, right, accelerate) yield similar Q-values—the state value is what matters. The dueling architecture can learn this shared value efficiently, focusing the advantage stream only on the states where action selection truly matters (e.g., approaching another car).

Prioritized Experience Replay

Standard experience replay samples transitions uniformly from the buffer. Prioritized experience replay (Schaul et al., 2016) instead samples transitions proportionally to their TD error:

$$p_j \propto \bigl|Q_\phi(s_j, a_j) - y_j\bigr|^\alpha,$$

where $\alpha$ controls the degree of prioritisation ($\alpha = 0$ reduces to uniform sampling). Transitions with large TD errors are "surprising"—the network's predictions are most wrong on these examples, so there is the most to learn from them.

Because prioritised sampling changes the data distribution, we must correct for this bias using importance sampling weights:

$$w_j = \left(\frac{1}{N \cdot p_j}\right)^\beta,$$

where $\beta$ is annealed from a small value to $1$ over training. These weights are applied to the loss function to ensure that the expected gradient remains unbiased.

Key Insight: Rainbow DQN

The Rainbow agent (Hessel et al., 2018) combines six DQN extensions: Double DQN, dueling architecture, prioritized replay, multi-step returns, distributional RL, and noisy networks. The combination is significantly better than any individual component, demonstrating that these improvements are largely complementary. Rainbow achieved state-of-the-art Atari performance at the time, dramatically outperforming vanilla DQN.

Q-Learning with Continuous Actions

The $\arg\max$ operation in Q-learning is trivial for discrete actions (just enumerate all actions), but becomes a difficult optimisation problem for continuous action spaces. Several approaches have been proposed:

Approach 1: Numerical Optimisation

Directly optimise $\max_a Q_\phi(s, a)$ using gradient ascent on $a$. This requires differentiating through the Q-network with respect to the action input. While conceptually simple, this is slow (requiring multiple gradient steps per action selection) and may find local optima.

Approach 2: Sampling-Based Methods

Use derivative-free optimisation such as the Cross-Entropy Method (CEM) or CMA-ES:

Sample $N$ actions from a distribution (e.g., Gaussian).
Evaluate $Q_\phi(s, a)$ for each.
Refit the distribution to the top-$k$ actions (elites).
Repeat for a few iterations.

CEM is surprisingly effective in practice and is used by several model-based RL methods. However, it scales poorly with action dimensionality.

Approach 3: Architectures with Tractable Maxima

Design the Q-network so that $\arg\max_a Q_\phi(s, a)$ has a closed-form solution. The Normalised Advantage Function (NAF) (Gu et al., 2016) parameterises the Q-function as a quadratic in $a$:

$$Q_\phi(s, a) = V_\psi(s) - \frac{1}{2}(a - \mu_\xi(s))^\top P_\xi(s) (a - \mu_\xi(s)),$$

where $P_\xi(s)$ is a positive-definite matrix. The maximum is trivially at $a^* = \mu_\xi(s)$. The drawback is that the quadratic form limits expressiveness—the Q-function may not capture complex, multimodal action-value landscapes.

Key Insight: Q-Learning vs. Actor-Critic for Continuous Control

For continuous action spaces, actor-critic methods (DDPG, TD3, SAC) are generally preferred over pure Q-learning because they avoid the difficult $\arg\max$ problem entirely—the actor network directly outputs actions. However, Q-learning retains advantages in discrete-action domains (board games, Atari, language) where the $\arg\max$ is trivial and maintaining a separate policy network is unnecessary overhead.

Practical Considerations

Exploration in Q-Learning

DQN uses $\epsilon$-greedy exploration: with probability $\epsilon$, take a random action; otherwise, take the greedy action $\arg\max_a Q_\phi(s, a)$. The exploration rate $\epsilon$ is typically annealed from 1.0 (fully random) to a small value (e.g., 0.01) over training. While simple, $\epsilon$-greedy can be inefficient in environments requiring directed exploration.

Replay Buffer Size

The buffer size trades off memory usage against data diversity. Too small a buffer leads to overfitting on recent transitions and catastrophic forgetting of earlier experience. Too large a buffer includes very stale data that may slow learning. For Atari, $10^6$ transitions (roughly 4 days of gameplay) is standard. For continuous control tasks, $10^5$ to $10^6$ is typical.

Target Network Update Frequency

With hard updates every $C$ steps, larger $C$ provides more stable targets but slower propagation of learned values. With soft updates (Polyak averaging), the parameter $\tau$ plays the analogous role. In either case, this hyperparameter significantly affects learning speed and stability.

Summary

This lecture covered Q-learning—the foundational value-based approach to deep RL. The key ideas are:

Q-learning learns $\Qstar$ directly and extracts the policy via $\arg\max$, eliminating the need for a separate actor network.
DQN made deep Q-learning practical through experience replay (breaking correlations, reusing data) and target networks (stabilising the moving-target regression).
The overestimation bias from the $\max$ operator is addressed by Double DQN, which decouples action selection from evaluation.
Dueling DQN separates state-value and advantage estimation, improving learning efficiency in states where the action choice matters little.
Prioritized experience replay focuses training on the most informative transitions.
For continuous actions, the $\arg\max$ is the main bottleneck; actor-critic methods generally dominate in this setting.

Q-learning and actor-critic represent two complementary paradigms in deep RL. Actor-critic methods are more natural for continuous control, while Q-learning excels in discrete-action domains. In the next lecture, we move to offline RL, where the agent must learn entirely from a fixed dataset without any further environment interaction—a setting where both paradigms face new challenges.