Lecture 14: Exploration

Why Is Exploration Hard?

Consider two Atari games. In Breakout, a random policy will occasionally hit the ball, score points, and receive reward. The agent can bootstrap from this accidental success, gradually improving. In Montezuma's Revenge, the agent must navigate multiple rooms, pick up a key, avoid enemies, and open doors—all before receiving any reward at all. A random policy will essentially never stumble upon the required sequence of actions. The first game is easy for RL; the second remains one of the hardest benchmarks in the field.

The difference comes down to exploration: how the agent discovers high-reward strategies in a vast space of possible behaviors. When reward is dense and immediately available, even simple exploration strategies (like adding random noise) suffice. When reward is sparse and requires temporally extended sequences of precise actions, exploration becomes the bottleneck.

Example

The Card Game Mao. Mao is a card game where the rules are secret—the only rule you may be told is "the only rule you may be told is this one." You incur a penalty when you break a rule, and you can only discover rules through trial and error. Rules do not always make sense intuitively. This is exactly the predicament of an RL agent: it must discover the structure of the environment (the rules) through interaction, with only sparse penalty signals to guide it. Now imagine that your goal in life was to win 50 games of Mao, and you did not know this in advance. Temporally extended tasks become increasingly difficult based on (1) how long the task is and (2) how little you know about the rules.

The Exploration-Exploitation Tradeoff

The exploration problem can be stated in two complementary ways:

How can an agent discover high-reward strategies that require a temporally extended sequence of complex behaviors that, individually, are not rewarding?
How can an agent decide whether to attempt new behaviors (to discover ones with higher reward) or continue to do the best thing it knows so far?

These are two facets of the same fundamental tradeoff:

Definition

Exploration vs. Exploitation.

Exploitation: Choosing the action that yields the highest expected reward according to the agent's current knowledge.
Exploration: Choosing actions the agent has not tried before (or has tried infrequently), in the hope of discovering strategies with even higher reward.

This tradeoff appears everywhere in decision making. In restaurant selection, exploitation means returning to your favorite restaurant, while exploration means trying a new one. In online advertising, exploitation means showing the most successful ad, while exploration means testing a different ad. In oil drilling, exploitation means drilling at the best known location, while exploration means drilling at a new one.

A Spectrum of Difficulty

Can we derive an optimal exploration strategy? The answer depends critically on the problem structure. Exploration problems fall along a spectrum of theoretical tractability:

Multi-armed bandits (one-step, stateless RL): Theoretically tractable. Optimal (up to constant factors) algorithms exist.
Contextual bandits (one-step RL with state): Tractable with additional structure (e.g., linear realizability).
Small, tabular MDPs: Tractable, with provably efficient algorithms that explore in polynomial time.
Large MDPs with known low-rank features: Sometimes tractable, depending on the specific structure.
Large MDPs without known structure: Generally intractable. No polynomial-time algorithm can guarantee efficient exploration in the worst case.

Deep RL problems with high-dimensional state spaces (images, proprioception) fall into the last category. This explains why principled exploration remains such a difficult open problem in practice: we lack the theoretical structure needed to guarantee efficient exploration, and must rely on heuristics, demonstrations, or meta-learning.

Exploration in Multi-Armed Bandits

To build intuition for principled exploration, we begin with the simplest possible setting: the multi-armed bandit, a one-step RL problem with no state transitions. The agent chooses one of $K$ arms (actions) at each round and receives a stochastic reward. The goal is to maximize cumulative reward over $T$ rounds.

Measuring Optimality: Regret

Definition

Regret. The regret after $T$ rounds of a bandit algorithm is the difference between the reward the agent would have obtained by always playing the best arm and the reward it actually obtained: $$\text{Reg}(T) = T \cdot \E[r(a^*)] - \sum_{t=1}^{T} \E[r(a_t)]$$ where $a^* = \arg\max_a \E[r(a)]$ is the optimal arm and $a_t$ is the action taken at round $t$. An algorithm with sublinear regret ($\text{Reg}(T) = o(T)$) is guaranteed to converge to the optimal arm.

Regret captures the total cost of exploration: the difference between what the agent could have earned with perfect knowledge and what it actually earned while learning. Lower regret means the agent identified and exploited the best arm more quickly.

Optimism in the Face of Uncertainty: UCB

The Upper Confidence Bound (UCB) algorithm embodies a simple but powerful principle: assume the unknown is good. For each arm, UCB maintains an estimate of the mean reward and a confidence interval around that estimate. It then selects the arm with the highest upper confidence bound—optimistically assuming that poorly explored arms might be better than they appear.

UCB1 (Upper Confidence Bound)

Initialize: Play each arm once. Set $N_a = 1$ and $\bar{r}_a = r_a$ for each arm $a$.
For $t = K+1, K+2, \ldots, T$:
Select action: $$a_t = \arg\max_a \left[\bar{r}_a + C\sqrt{\frac{\ln t}{N_a}}\right]$$ where $\bar{r}_a$ is the empirical mean reward of arm $a$, $N_a$ is the number of times arm $a$ has been pulled, and $C > 0$ is a constant.
Observe reward $r_t$. Update $N_{a_t} \leftarrow N_{a_t} + 1$ and $\bar{r}_{a_t}$.

UCB1 achieves regret $O(\sqrt{KT \ln T})$, which is optimal up to logarithmic factors.

The confidence bonus $C\sqrt{\ln t / N_a}$ decreases as an arm is pulled more often (larger $N_a$) and increases as total time grows (larger $t$). Arms that have been explored less receive a larger bonus, encouraging the algorithm to try them. As the algorithm collects more data, the confidence intervals shrink, and it increasingly exploits the best arm.

Key Insight

UCB's optimism principle provides a natural resolution to the exploration-exploitation dilemma: by assuming that uncertain options are as good as they plausibly could be, the algorithm automatically explores under-sampled arms while gravitating toward the best arm as uncertainty is resolved. This "optimism in the face of uncertainty" principle extends to more complex settings, including MDPs, where it underlies algorithms like UCB-VI.

Probability Matching: Posterior Sampling

An alternative to optimism is posterior sampling (also called Thompson sampling), which uses a Bayesian approach. The agent maintains a posterior distribution over the reward parameters of each arm and acts by sampling from this posterior.

Thompson Sampling for Bandits

Assume a model for each arm's reward: $r(a_i) \sim p_{\psi_i}(r)$ for each arm $a_i$.
Maintain a posterior belief $\hat{p}(\psi_1, \ldots, \psi_K)$ over the reward parameters, initialized with a prior.
For each round $t = 1, 2, \ldots, T$:
(a) Sample reward parameters $\psi_1, \ldots, \psi_K \sim \hat{p}(\cdot)$ from the current posterior.
(b) Take the optimal action assuming the sampled parameters are correct: $a_t = \arg\max_a \E_{p_{\psi_a}}[r]$.
(c) Observe reward $r_t$ and update the posterior $\hat{p}$ using Bayes' rule.

Thompson sampling achieves Bayes-optimal regret and is competitive with UCB in practice. For Bernoulli bandits with a Beta prior, the posterior update is analytically tractable.

The intuition behind Thompson sampling is elegant: by sampling from the posterior, the algorithm naturally explores arms about which it is uncertain (since there is variance in the sampled parameters) while exploiting arms it believes are good (since the posterior concentrates around the true value). As more data is collected, the posterior sharpens, and the sampled parameters increasingly agree with reality, leading to exploitation.

Key Insight

UCB and Thompson sampling represent two complementary philosophies for principled exploration: optimism (assume the unknown is good) versus probability matching (act according to your beliefs). Both achieve near-optimal regret bounds for bandits. Thompson sampling is often easier to implement and tends to perform better empirically, while UCB provides tighter worst-case guarantees. Both methods are widely deployed in industry for recommender systems and ad placement.

Exploration in Large MDPs

The elegant theory of bandit exploration does not easily translate to the large MDPs that arise in deep RL. In an MDP, the agent's actions affect not only the immediate reward but also the future state, creating long-range dependencies that bandits do not have. The state space may be continuous and high-dimensional (e.g., images), making count-based methods inapplicable in their basic form.

There has been substantial research on extending principled exploration to large MDPs—count-based exploration with density models, curiosity-driven exploration via prediction error, and random network distillation, among others. However, a sobering practical reality emerges:

Key Insight

At the end of the day, exploration from scratch in large MDPs is intractable in the worst case. The most successful practical approaches for complex tasks sidestep the pure exploration problem entirely by relying on: (1) demonstrations or base models pretrained on demonstrations, which provide a warm start in the policy space, and (2) shaped rewards wherever feasible, such as hand-coded reward shaping in simulation for legged robots or single-step rewards from human preferences for language models.

This pragmatic observation motivates the second half of this lecture: can we use meta-learning to learn effective exploration strategies from a distribution of tasks, rather than designing them by hand?

Learning to Explore via Meta-Learning

In Lecture 13, we introduced meta-RL and identified the central challenge: the agent must explore effectively at test time to identify the task and then execute well. We now examine this exploration challenge in depth and present a solution that elegantly decouples the learning of exploration from the learning of execution.

When Do We Need Test-Time Exploration?

Test-time exploration is essential in settings with partial observability—where the agent cannot determine the task from a single observation:

Physical environments: A robot in a new kitchen must open cabinets to find ingredients. The task (what to cook) is partially observable until the robot actively explores.
LLM reasoning: A language model solving a complex problem must explore different reasoning strategies to identify which approach works. The "task" (which solution strategy is correct) is not immediately apparent.
Game playing: Facing a new opponent, the agent must probe the opponent's strategy through trial interactions before it can respond optimally.

The Chicken-and-Egg Problem Revisited

As discussed in Lecture 13, end-to-end meta-RL methods face a coupling problem. Learning exploration and execution depend on each other, which can lead to poor local optima and poor sample efficiency. To make this concrete, consider the following scenario:

A meta-RL agent is learning to cook in different kitchens. The exploration phase involves finding ingredients, and the execution phase involves cooking the recipe. If the agent never learns to find the right ingredients (bad exploration), it cannot learn to cook (bad execution). If it cannot cook, it receives low reward regardless of how well it explores, providing no gradient signal to improve exploration. This coupling traps the optimization.

DREAM: Decoupled Exploration and Execution

The DREAM algorithm (Decoupled Reward-free Exploration and Execution in Meta-Reinforcement Learning; Liu, Raghunathan, Liang, Finn, ICML 2021) provides an elegant solution to the chicken-and-egg problem. The core idea is to separate meta-training into two independent optimization problems.

Definition

DREAM: Decoupled Exploration and Execution. DREAM separates meta-training into two phases:

Phase 1 — Learn execution and identify key information: Given the task identifier $\mu_i$ (available during meta-training), learn a bottlenecked representation $z_i = F(\mu_i)$ and an execution policy $\pi^{\text{exec}}(a \mid s, z_i)$. The bottleneck ensures that $z_i$ captures only task-relevant information.
Phase 2 — Learn to explore: Train an exploration policy $\pi^{\text{exp}}$ to collect data $\mathcal{D}_{\text{tr}}$ such that $z_i$ can be accurately recovered from $\mathcal{D}_{\text{tr}}$ alone.

At meta-test time: (1) explore using $\pi^{\text{exp}}$ to collect $\mathcal{D}_{\text{tr}}$, (2) infer $\hat{z} \sim q(z \mid \mathcal{D}_{\text{tr}})$, and (3) execute using $\pi^{\text{exec}}(a \mid s, \hat{z})$.

The decoupling is the key design choice. Phase 1 determines what information the agent needs (the bottlenecked representation $z_i$), while Phase 2 determines how to gather that information (the exploration policy). Because these are separate optimization problems, neither depends on the other being already solved. This breaks the chicken-and-egg cycle.

The Variational Information Bottleneck

A critical component of DREAM is the information bottleneck that ensures the representation $z_i$ captures only task-relevant information. Without this bottleneck, the representation might encode irrelevant details (wall colors, decorations) that are easy to observe but useless for task execution. The exploration policy would then waste effort gathering this irrelevant information.

The bottleneck is implemented using a variational information bottleneck (VIB). The idea is to add Gaussian noise to the representation and prevent the encoder from circumventing the noise by increasing the representation magnitude:

The encoder produces $z_i = F(\mu_i)$.
During training, add Gaussian noise: $\bar{z}_i = z_i + \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$.
Regularize the representation magnitude via a KL penalty: $\mathcal{L}_{\text{VIB}} = \mathcal{L}_{\text{task}} + \beta \|z_i\|^2$.

This is equivalent to minimizing $D_{\text{KL}}(F(z \mid \mu_i) \| \mathcal{N}(0, I))$, forcing the representation to be compact and informative. The noise destroys any information not strongly encoded in the representation, while the $\ell_2$ penalty prevents the encoder from simply scaling up the signal to overwhelm the noise.

Key Insight

The information bottleneck serves a dual purpose: it ensures the execution policy relies only on genuinely task-relevant features, and it defines a clear target for the exploration policy—recover a compact, low-dimensional representation. Without the bottleneck, the exploration policy would face the impossible task of reconstructing the full task identifier, which might include vast amounts of irrelevant detail.

The Exploration Reward

How should we train the exploration policy $\pi^{\text{exp}}$ to collect informative data? DREAM uses an information gain reward: the per-step increase in the exploration policy's ability to predict $z_i$.

Specifically, DREAM trains a task inference model $q(z_i \mid \mathcal{D}_{\text{tr}})$ and defines the exploration reward as:

$$r_t^{\text{exp}} = \underbrace{\ell(q(z_i \mid \tau_{1:t-1}))}_{\text{prediction error from past data}} - \underbrace{\ell(q(z_i \mid \tau_{1:t}))}_{\text{prediction error after new observation}}$$

where $\ell$ is a prediction loss (e.g., mean squared error in predicting $z_i$). This reward is positive when the new observation at time $t$ reduces uncertainty about the task representation, encouraging the agent to seek informative interactions.

Theoretical Analysis

DREAM's decoupled objective enjoys theoretical properties that end-to-end meta-RL methods lack.

Theorem

Consistency of DREAM (Informal). Under mild assumptions, the DREAM objective is consistent with end-to-end optimization: the optimal solution to the decoupled problem coincides with the optimal solution to the end-to-end meta-RL objective. That is, DREAM does not sacrifice optimality through decoupling—it can in principle recover the optimal exploration strategy.

Furthermore, in a simplified bandit-like analysis with $|A|$ arms, the sample complexity comparison between end-to-end meta-RL and DREAM is striking:

RL$^2$ (end-to-end): Requires $O(|A|^2 \log |A|)$ meta-training samples to learn optimal exploration.
DREAM (decoupled): Requires $O(|A| \log |A|)$ meta-training samples—a factor-of-$|A|$ improvement.

This quadratic-to-linear improvement arises because DREAM does not need to simultaneously learn what information is relevant and how to gather it. By separating these two problems, each becomes individually easier.

Empirical Results

DREAM was evaluated on a challenging sparse-reward 3D visual navigation problem. The agent must navigate a 3D environment using pixel observations (80 x 60 RGB images), read a sign indicating the target object (key, block, or ball of a specific color), and then navigate to the correct object. The reward is sparse and binary: $+1$ if the correct object is reached, $0$ otherwise.

Qualitative Behavior

The exploration and execution behaviors learned by DREAM are intuitive. During the exploration episode, the agent walks around the barrier to read the sign, identifying the task. During the execution episode, it proceeds directly to the correct object based on the information gathered during exploration. This demonstrates that DREAM learns a semantically meaningful exploration strategy: it identifies what information is needed (the sign content) and executes a targeted plan to acquire that information.

Quantitative Comparison

Compared against several baselines:

End-to-end methods (RL$^2$, IMPORT, VariBAD): Perform poorly due to the exploration-execution coupling problem. They struggle to learn meaningful exploration strategies in the sparse-reward setting.
PEARL with Thompson sampling: An upper-bound variant (PEARL-UB) using the optimal policy with Thompson sampling exploration does not learn the optimal exploration strategy—it cannot discover that reading the sign is the most efficient way to identify the task.
DREAM: Achieves near-optimal reward, successfully learning both what to explore (the sign) and how to execute (navigate to the target).

Example

Comparison Summary. On the 3D visual navigation task, DREAM achieves near-optimal success rate, while end-to-end methods (RL$^2$, IMPORT, VariBAD) fail to learn meaningful exploration and PEARL's Thompson sampling exploration is suboptimal because it does not learn that reading the sign is the most information-efficient exploration action.

Application: Meta-Exploration for CS Education

An unexpected application of meta-exploration comes from computer science education. The task of grading and providing feedback on student programming assignments can be framed as a meta-RL problem where each student's program is a different "task."

Finding Bugs and Providing Feedback

Consider a programming assignment where students write code for a simple game (e.g., the Bounce assignment on Code.org, or the Breakout assignment in Stanford's CS106A). Each student's program may contain different bugs. The meta-RL agent must:

Explore: Interact with the student's program by providing inputs and observing outputs to identify the specific bugs present.
Execute: Generate an accurate assessment of the program's correctness based on the observed behavior.

The meta-exploration framework applies directly. During meta-training, the agent learns an exploration strategy that efficiently tests different aspects of a program (what happens when the ball hits the goal? the floor? the wall?). At meta-test time, it applies this learned strategy to a new student's program and uses the gathered information to assign grades or provide targeted feedback.

AI-Assisted Grading

Liu et al. (SIGCSE 2024) deployed this approach in Stanford's CS106A course for the Breakout programming assignment. An autograder pre-populates a rubric and shows videos of the learned test inputs applied to each student's program. In a controlled study with Stanford TAs:

AI-assisted grading was 44% faster than manual grading.
AI-assisted grading was 6% more accurate than manual grading.
TAs reported positive experiences using the system.

This application illustrates the breadth of meta-exploration: the same framework that learns to explore mazes or identify object properties can learn to systematically test software, with practical impact on education at scale.

Looking Ahead

This lecture covered the exploration problem from first principles to frontier research. We began with the fundamental exploration-exploitation tradeoff, studied principled bandit algorithms (UCB and Thompson sampling), and confronted the intractability of exploration in large MDPs. We then showed how meta-learning can learn exploration strategies, with DREAM providing an elegant decoupled approach that avoids the chicken-and-egg problem.

Key takeaways:

Exploration is the bottleneck in sparse-reward RL: without it, the agent never discovers high-reward strategies.
In bandits, UCB (optimism) and Thompson sampling (probability matching) provide near-optimal exploration with theoretical guarantees.
In large MDPs, exploration from scratch is generally intractable; practical approaches rely on demonstrations, shaped rewards, or meta-learning.
DREAM decouples exploration and execution learning via an information bottleneck, achieving near-optimal meta-exploration with better sample complexity than end-to-end methods.
Meta-exploration has surprising applications beyond robotics, including automated assessment of student programs.

In the next lecture, we turn to hierarchical RL, which addresses a related challenge: can we decompose long-horizon tasks into a hierarchy of subtasks, enabling the agent to plan and explore at multiple levels of temporal abstraction?