Probably Approximately Correct (PAC) Learning

Over the past two lectures, we built two families of exploration algorithms for bandits: frequentist methods like UCB and Bayesian methods like Thompson Sampling. Both measure performance through regret—the cumulative gap between our rewards and the best arm's rewards over $T$ rounds.

But regret hides something important. An algorithm with $O(\sqrt{T})$ regret could achieve that by making many tiny mistakes—or by making a few catastrophic ones. In most settings we don't care which. But imagine you're a surgeon relying on an RL system to recommend treatments. You don't just want good average performance—you want a guarantee that after some learning period, the system almost never recommends a seriously bad action.

This motivates Probably Approximately Correct (PAC) learning. Instead of bounding cumulative regret, PAC bounds the number of time steps on which the agent acts significantly suboptimally. The "probably" captures a failure probability $\delta$, and the "approximately" captures a tolerance $\epsilon$.

Where does PAC fit among the evaluation criteria we've seen so far? Think of it as a ladder, each rung more demanding:

1. Empirical evaluation — run it and measure
2. Asymptotic convergence — does it eventually find the optimal policy?
3. Regret bounds — how does total suboptimality grow with $T$?
4. PAC bounds — after how many steps does the agent stop making non-trivial mistakes?

Notice that PAC and regret are complementary, not competing. An algorithm can have low regret but fail PAC (by concentrating errors into rare catastrophic steps), or be PAC but have moderate regret (by taking many slightly suboptimal steps during learning). Most PAC algorithms use optimism or Thompson sampling—some simply initialize all values high to encourage exploration.

Fast RL in MDPs

Now let's make the leap from bandits to full MDPs. The same principles carry over—optimism, probability matching, PAC—but the complexity jumps considerably. In a bandit, pulling an arm tells you about that arm's reward. In an MDP, taking action $a$ at state $s$ reveals both the reward $R(s,a)$ and the transition dynamics $T(s'|s,a)$. Both matter for long-term planning.

Even more challenging: reaching a particular state-action pair may require executing a multi-step strategy. Exploration in MDPs isn't just about trying different actions—it involves planning to explore.

The PAC-MDP formulation captures this. Instead of requiring each action to be $\epsilon$-optimal in immediate reward, we require the agent's policy value at the current state to be close to optimal:

The key question becomes: how quickly can an algorithm achieve PAC-MDP status? The answer depends on how efficiently it explores the state-action space, and the algorithms we develop in this lecture will achieve polynomial sample complexity in all problem parameters.

Model-Based Interval Estimation with Exploration Bonus (MBIE-EB)

Remember how UCB works in bandits? We add a confidence bonus to each arm's estimated reward, making under-explored arms look optimistically good. MBIE-EB (Strehl and Littman, 2008) does the same thing for MDPs: maintain a model, but inflate the reward at under-visited state-action pairs with an exploration bonus. It's UCB's optimism principle, transplanted into the Bellman equation.

The agent tracks visit counts $n_{sa}$ for each state-action pair and transition counts $n_{sas'}$ for each triple. From these, it builds empirical estimates:

Instead of acting greedily on this model, MBIE-EB adds an exploration bonus $\frac{\beta}{\sqrt{n_{sa}}}$ to each state-action pair's reward and solves the augmented MDP. The bonus shrinks as $1/\sqrt{n}$—rarely-visited states look optimistically good (explore), while well-visited states rely on their true estimates (exploit). The key: this bonus lives inside the Bellman backup:

where the exploration coefficient is:

The agent acts greedily on $\tilde{Q}$: pick $a_t = \arg\max_{a} \tilde{Q}(s_t, a)$, observe the transition, update counts, and re-solve.

Notice that Q-values start at $\frac{1}{1-\gamma}$—the maximum possible value when rewards are in $[0, 1]$. This optimistic initialization gives unexplored pairs a built-in appeal, providing a second source of exploration pressure on top of the explicit bonus.

The Simulation Lemma

Here's a question that seems obvious but turns out to be subtle: if we learn an approximate model of an MDP—getting rewards and transitions mostly right—does that mean our policy will be good? Not necessarily! Small model errors could compound over a long horizon and produce terrible decisions.

The Simulation Lemma tells us exactly when we're safe. It bridges model accuracy and policy quality: if $\hat{T} \approx T$ and $\hat{R} \approx R$, then $V^{\pi}$ in the approximate model is close to $V^{\pi}$ in the true one. This is the theoretical backbone of MBIE-EB's PAC guarantee.

Notice the $(1 - \gamma)$ in the denominator. Errors at each step propagate through the entire horizon, so longer horizons (larger $\gamma$) amplify model errors. This is intuitive: if you're planning a one-step trip, a slightly wrong map barely matters. If you're planning a cross-country journey, small errors accumulate.

MBIE-EB is PAC

We can now put the pieces together: MBIE-EB is PAC-MDP. The proof has three ingredients:

1. Concentration — after $m$ visits to a state-action pair, $\hat{R}$ and $\hat{T}$ are close to the true values with high probability.
2. Simulation Lemma — accurate models yield near-optimal policies.
3. Counting argument — there are $|\mathcal{S}| \times |\mathcal{A}|$ pairs, and each can be "under-visited" at most $m$ times. So the total number of bad steps is at most $|\mathcal{S}| \cdot |\mathcal{A}| \cdot m$.

On all other steps, the model is accurate enough and the policy is near-optimal. That's the PAC guarantee.

The sample complexity is polynomial in all problem parameters—this is what makes MBIE-EB a PAC-MDP algorithm. The dominant term scales as $\tilde{O}\!\left(\frac{|\mathcal{S}|^2 |\mathcal{A}|}{\epsilon^2 (1-\gamma)^4}\right)$, reflecting the cost of learning transitions for all $|\mathcal{S}||\mathcal{A}|$ state-action pairs to accuracy $\epsilon$ over an effective horizon of $\frac{1}{1-\gamma}$.

Bayesian Model-Based RL

In Lecture 11, we saw how Bayesian bandits maintain a posterior over reward parameters and explore via Thompson Sampling. The natural next step: maintain a posterior over the entire MDP—transitions $\mathcal{P}$ and rewards $\mathcal{R}$.

The posterior $p(\mathcal{P}, \mathcal{R} \mid h_t)$ updates as we observe states, actions, and rewards. With Dirichlet priors over transitions (the conjugate for categorical distributions), these updates stay analytically tractable—just like Beta-Bernoulli for bandits.

The same two exploration strategies carry over:

• Bayesian UCB: Construct confidence bounds from posterior quantiles and act optimistically.
• Thompson Sampling (Probability Matching): Sample an MDP from the posterior and act optimally under the sampled model.

The conceptual leap is straightforward, but there's an important practical difference: "acting optimally under a sample" now means solving an entire MDP, not just picking the arm with the highest mean. That computational cost is the main challenge.

The upside? If the prior is well-calibrated, Bayesian methods can be far more data-efficient than frequentist ones. The prior encodes structural knowledge—like "nearby states have similar dynamics"—that pure counting methods like MBIE-EB cannot exploit.

Posterior Sampling for Reinforcement Learning (PSRL)

PSRL (Osband, Russo, and Van Roy, 2013) is beautifully simple. At the start of each episode: sample a complete MDP from the posterior, solve it to get the optimal policy, and follow that policy for the entire episode. Then update the posterior and repeat.

Just like Thompson Sampling for bandits—but instead of sampling a reward and picking the best arm, we sample an entire world model and plan optimally within it. The probability matching property still holds:

Why sample per-episode instead of per-step? This is crucial for deep exploration. If the posterior says a distant state might be highly rewarding, PSRL commits to a policy that navigates there. Compare this to $\epsilon$-greedy, which explores locally but can never execute a multi-step plan to reach an informative state far away.

Generalization and Strategic Exploration

Everything so far—MBIE-EB, PSRL, PAC—treats each state-action pair independently. That's fine when $|\mathcal{S}|$ and $|\mathcal{A}|$ are small. But in Atari, the "state" is a screen of pixels ($256^{210 \times 160 \times 3}$ possible images). In robotics, states are continuous joint angles. We'll never visit most states even once, let alone the multiple times MBIE-EB needs.

We need generalization: using experience from similar states to inform behavior at novel ones. The visit-count approach completely breaks down here.

The bridge between tabular methods and large-scale RL is the contextual bandit. Before choosing an action, the agent observes a context (feature vector) $s$, and the reward depends on both context and action.

The payoff of generalization is dramatic. LinUCB replaces raw visit counts with confidence ellipsoids from linear regression. As the number of arms $K$ grows, UCB's regret scales linearly with $K$ (each arm explored independently), while LinUCB stays nearly flat—learning about one arm immediately constrains all others through the shared feature space.

The key shift: instead of asking "how many times have I visited this state?", we ask "how uncertain is my model about this state, given everything I've seen?" A state we've never visited can still have low uncertainty if it's similar (in feature space) to states we know well.

Exploration Bonuses for Deep RL

Can we bring MBIE-EB's optimism into deep RL? The idea is simple: add an exploration bonus $r_{\text{bonus}}(s, a)$ to the Q-learning update, approximating what count-based bonuses do in the tabular setting.

The standard Q-learning weight update with function approximation is:

With an exploration bonus, this becomes:

But how do you count "visits" in a continuous state space? You can't. Instead, researchers have found creative proxies for novelty:

• Count-based exploration (Bellemare et al., NIPS 2016): Use density models to compute pseudo-counts—an effective visit count derived from how much a state changes the density model's predictions.
• Hash-based counts (Tang et al., NIPS 2017): Hash high-dimensional states into a lower-dimensional space and count visits in hash buckets. Simple but surprisingly effective.
• Neural density models (Ostrovski et al., ICML 2017): Extend count-based exploration to high-dimensional spaces using PixelCNN density models to derive pseudo-counts, enabling effective exploration in visually rich environments like Atari.

One important caveat: bonus terms are computed at the time of visit. During experience replay, a state that was novel when first visited may have been seen many times by the time the replay happens—making the stored bonus stale.

Thompson Sampling with Function Approximation

Thompson Sampling faces a harder challenge with neural networks: maintaining a posterior over millions of weights is intractable. Three pragmatic approximations have emerged:

• Sampling over representations (Mandel et al., 2016) — sample both features and parameters, exploring over model structure, not just fixed-architecture weights.
• Bootstrapped DQN (Osband et al., 2016) — train $C$ independent DQN "heads" on bootstrapped replay samples. Each episode, pick a random head and follow it. The diversity among heads proxies for posterior uncertainty—no explicit Bayesian inference needed. Exploration is relatively shallow, though.
• Bayesian Deep Q-Networks (Azizzadenesheli and Anandkumar, 2017) — use a neural net for features but apply Bayesian linear regression on the last layer. This gives a tractable posterior with principled uncertainty, and empirically outperforms bootstrapping.

The gap between theory (exact posteriors give optimal exploration) and practice (approximate posteriors give heuristic exploration) remains open. No current method achieves both the computational efficiency of deep networks and the guarantees of exact Bayesian inference.

Meta-Learning for RL Exploration

Every exploration method so far has been hand-designed: a researcher specifies the strategy (add a bonus, sample from a posterior) and proves it works. But here's a provocative question: can agents learn to explore?

The meta-learning approach trains agents across many tasks so they internalize good exploration behavior. Instead of being told to add $\beta / \sqrt{n}$, the agent discovers from experience what exploration patterns lead to fast learning.

DREAM

DREAM (Liu et al., 2022) trains a policy to explore effectively across a distribution of tasks, learning strategies that generalize to new environments.

Decision-Pretrained Transformer (DPT)

DPT (Lee, Xie, Pacchiano, Chandak, Finn, Nachum, Brunskill, 2023) takes a different approach: train a transformer on trajectories of $(s, a, r, s')$ tuples. Given past experience $D$ and a query state $s_{\text{query}}$, the transformer outputs an action distribution $M_\theta(a^* \mid s_{\text{query}}, D)$.

Why does this work? Training to predict optimal actions given a dataset implicitly mimics Thompson Sampling—but with a learned prior instead of a hand-specified one. Thompson Sampling is optimal when the prior is correct, but choosing the right prior in complex domains is itself an unsolved problem. DPT learns its implicit prior from data across many tasks, and the transformer architecture can represent far richer task distributions than any analytically tractable prior family.

The results are striking: on linear bandits, DPT outperforms both Thompson Sampling and LinUCB. Learned exploration can surpass hand-designed algorithms when task structure is available in training data.

Theoretical Results and Open Problems

Where does the theory stand? For tabular MDPs, it's essentially complete:

• Regret — tight minimax bounds (Azar, Osband, Munos, 2017)
• PAC — tight bounds (Dann, Li, Wei, Brunskill, 2019)
• Instance-dependent — bounds that adapt to the specific problem, not worst-case (Zanette and Brunskill, 2019; Simchowitz and Jamieson, 2019)

The frontier has shifted to function approximation. Jin et al. (2020) gave the first provably efficient algorithms for RL with linear function approximation. But the deeper question—what makes exploration hard?—requires entirely new complexity measures like eluder dimension (Russo and Van Roy) and Bellman rank (Jiang et al.).

A unified characterization remains elusive. This is one of the most active areas in RL theory, and progress here would directly impact practical deep RL systems.

Summary

This lecture completed our three-lecture arc on data-efficient RL (Lectures 10–12), extending exploration from bandits to full MDPs and beyond.

We introduced PAC as a complementary evaluation criterion to regret—especially relevant in safety-critical settings where we care about bounding the count of bad decisions. MBIE-EB achieves PAC-MDP guarantees by injecting optimistic exploration bonuses into the Bellman backup, grounded by the Simulation Lemma connecting model accuracy to policy quality.

On the Bayesian side, PSRL extends Thompson Sampling to MDPs with episode-level commitment that enables deep exploration. When tabular methods hit their limits, contextual bandits and LinUCB replace visit counts with model uncertainty, and deep RL exploration bonuses bring intrinsic motivation to neural networks. Finally, meta-learning approaches like DPT can learn to explore, outperforming hand-designed algorithms.

The algorithm-to-guarantee mapping for the full exploration arc: