From Multi-Task Learning to Meta-Learning

In the previous lecture, we trained a single policy to handle a distribution of tasks by conditioning on a task identifier $z_i$. This works well when the task identifier is available at test time—for instance, when a human provides a language instruction. But what happens when the agent encounters a new task for which no explicit identifier is given? The agent must figure out what the task is from interaction alone, and it must do so quickly.

This is the domain of meta-reinforcement learning (meta-RL): training agents that can learn to learn, adapting to new tasks from a small amount of experience. Where multi-task RL asks "can we train one policy for many tasks?", meta-RL asks the deeper question: "can we train an agent that rapidly adapts to any task drawn from a distribution, using only a handful of episodes?"

The Transfer Learning Spectrum

Meta-RL sits on a spectrum of approaches that leverage shared structure across tasks:

• Forward transfer: Train on one or more source tasks, then fine-tune on a target task. The source training provides a good initialization, but adaptation still requires substantial data. Example: pretraining a vision backbone on ImageNet, then fine-tuning for a downstream task.
• Multi-task learning: Train on all tasks jointly, producing a single model that handles them all. As discussed in Lecture 12, this requires an explicit task identifier at test time.
• Meta-learning: Train on a distribution of tasks such that the resulting system can adapt to a new task from very few examples. The adaptation procedure itself is learned during training.

Meta-learning is the most ambitious of these: it asks the training process to discover not just good features or policies, but a good learning procedure.

The Meta-Learning Problem Setup

Meta-learning has been widely studied in supervised learning under the umbrella of few-shot learning. The setup there provides useful intuition for the RL case.

Few-Shot Learning in Supervised Settings

In few-shot image classification, the agent must classify images of new categories after seeing only a handful of labeled examples per category. The standard protocol involves:

1. Meta-training: Present the learner with many "episodes," each consisting of a small support set (a few labeled examples of novel classes) and a query set (test images to classify).
2. Meta-testing: Evaluate on entirely new classes not seen during meta-training, again with only a few examples per class.

The meta-learner is trained end-to-end across episodes so that its adaptation mechanism (whether gradient-based, metric-based, or memory-based) becomes effective at few-shot classification.

Meta-RL Problem Formulation

We now translate this to reinforcement learning. We have a distribution over tasks $p(\mathcal{M})$, where each task $\mathcal{M}_i$ is an MDP. The meta-RL protocol mirrors the few-shot setup:

The critical distinction from multi-task RL is that the agent receives no explicit task identifier $z_i$. Instead, it must infer the task identity from its exploration experience. The exploration episodes serve as the "support set," and the execution episode as the "query set," drawing a direct parallel to few-shot supervised learning.

Examples of Task Distributions

Task distributions arise naturally in many settings:

• Locomotion: Walking across different terrain types (ice, sand, slopes) where the dynamics change.
• Manipulation: Picking up objects of varying shapes, sizes, and weights.
• Navigation: Finding goals in different environments.
• Dialogue: Conversing with users who have different preferences or needs.
• Game playing: Adapting strategy when facing opponents with different play styles.

Black-Box Meta-RL

The most conceptually simple approach to meta-RL is the black-box (or context-based) method: train a neural network with memory that takes in the entire history of interactions and outputs actions. The adaptation to a new task happens implicitly within the network's forward pass—the memory accumulates information about the task, and the policy conditions on this accumulated context.

Training Black-Box Meta-RL

The training procedure for black-box meta-RL is straightforward:

At test time, the procedure is identical except that we freeze $\theta$. The agent encounters a new task, and its internal memory state evolves as it interacts, effectively performing task inference and policy adaptation through the learned dynamics of the recurrent network.

Architectures for Black-Box Meta-RL

Several neural network architectures have been proposed for black-box meta-RL, each offering different tradeoffs in terms of memory capacity, computational efficiency, and the types of adaptation they can express:

• RL$^2$ (Duan et al., 2016; Wang et al., 2016): Uses an RNN (LSTM or GRU) as the meta-learner. The RNN's hidden state accumulates information across time steps and episodes. At each step, it receives the current observation, previous action, and previous reward as input. The RNN's recurrence naturally implements a form of Bayesian updating, though the specific update rule is learned rather than hand-designed.
• SNAIL (Mishra et al., 2018): Uses a combination of temporal convolutions and self-attention layers. Temporal convolutions provide efficient aggregation over long sequences, while attention allows the model to selectively focus on the most relevant past experiences. This architecture can scale to longer adaptation horizons than vanilla RNNs.
• PEARL (Rakelly et al., 2019): Uses a feedforward encoder that processes individual transitions $(s, a, r, s')$ independently, then averages the resulting embeddings to form a task context vector $z$. The policy conditions on this context: $\pi(a \mid s, z)$. This permutation-invariant aggregation is computationally efficient and works well when the order of exploration transitions is not critical for task identification.

Examples and Capabilities

Black-box meta-RL agents can exhibit remarkably sophisticated behavior. In maze navigation tasks, an RL$^2$ agent learns to systematically explore corridors it has not visited, building an internal map of the maze layout. When placed in a new maze, it navigates to the goal far more efficiently than an agent trained from scratch. In continuous control tasks, PEARL adapts its locomotion strategy within a few episodes when the dynamics change—for instance, when a leg becomes injured or the terrain properties shift.

Perhaps most strikingly, black-box meta-RL has connections to the way modern large language models reason. Systems like DeepSeek R1 use chain-of-thought reasoning that can be viewed as a form of in-context adaptation: the model processes the problem description (analogous to exploration episodes) and then generates a solution (the execution episode), with the "adaptation" happening entirely within the forward pass of the transformer. Both are instances of using the meta-learned network dynamics to implement a learned algorithm in the activations.

Connection to Multi-Task Policies

There is a deep connection between black-box meta-RL and the multi-task policies from Lecture 12. Recall that a multi-task policy conditions on a task identifier: $\pi(a \mid s, z_i)$. In black-box meta-RL, the accumulated experience $\mathcal{D}_{\text{tr}} = \{(s_t, a_t, r_t, s_{t+1})\}$ from exploration episodes serves as an implicit task identifier.

Consider the PEARL architecture specifically. It computes a context vector $z = \frac{1}{|\mathcal{D}_{\text{tr}}|} \sum_{(s, a, r, s') \in \mathcal{D}_{\text{tr}}} q_\phi(s, a, r, s')$, where $q_\phi$ is a learned encoder. The policy then acts as $\pi(a \mid s, z)$—which is exactly a multi-task policy conditioned on a learned task embedding. The difference from Lecture 12 is that $z$ is inferred from experience rather than given as input.

Meta-RL as a POMDP

A powerful theoretical perspective views meta-RL through the lens of partially observable Markov decision processes (POMDPs). The task identity $\mathcal{M}_i$ is a hidden variable that the agent cannot directly observe. Instead, the agent receives observations, rewards, and transitions that are generated by the unknown task. The agent must maintain a belief over which task it is in and act optimally given this uncertainty.

Formally, we can construct a POMDP where:

• The hidden state includes the task identity $\mathcal{M}_i$ (which is fixed within an episode but unknown to the agent).
• The observations are the standard state observations $s_t$ and rewards $r_t$, which provide partial information about the hidden task.
• The belief state $b_t = p(\mathcal{M}_i \mid h_t)$ is the agent's posterior distribution over tasks given the interaction history $h_t$.

Under this POMDP view, the optimal meta-RL policy is the Bayes-optimal policy for this POMDP. It maintains a belief over tasks and takes actions that are optimal given the current belief—including actions that are informative (exploration) rather than immediately rewarding (exploitation).

This Bayes-optimal perspective clarifies why exploration and exploitation must be interleaved: each exploration action has both an immediate cost (potentially suboptimal reward) and an information-gathering benefit (reducing task uncertainty for future decisions).

The Exploration Challenge in Meta-RL

Exploration is the central challenge that distinguishes meta-RL from multi-task RL. In multi-task RL, the task identifier is given explicitly. In meta-RL, the agent must discover the task through exploration, and the quality of exploration directly determines the quality of execution. This creates a difficult optimization landscape.

The Chicken-and-Egg Problem

End-to-end meta-RL training faces a fundamental coupling problem between exploration and execution:

• If exploration is poor, the agent gathers uninformative data, making good execution impossible.
• If execution is poor, the gradient signal for improving exploration is weak or absent, since even good exploration data cannot rescue a bad execution policy.

This chicken-and-egg problem can trap the meta-learner in poor local optima. Consider a cooking robot that must first find ingredients (exploration) and then cook a dish (execution). If the robot never learns to find the right ingredients, it cannot learn to cook, and if it cannot cook, it has no incentive to search for ingredients more carefully. End-to-end training must simultaneously improve both, which is challenging when the reward signal is sparse.

Alternative Exploration Strategies

Several approaches have been proposed to sidestep the chicken-and-egg problem by decoupling exploration from end-to-end training:

• Posterior sampling (Thompson sampling): PEARL uses this classical idea. The agent maintains a distribution $q(z \mid \mathcal{D}_{\text{tr}})$ over task embeddings, samples $z$ from the current posterior, and acts optimally for that sampled task. This provides principled exploration without requiring end-to-end optimization of the exploration strategy. However, it can be suboptimal by an arbitrarily large amount in some environments where more targeted information gathering is needed.
• Intrinsic reward methods: MAME (Gurumurthy et al., 2019) adds intrinsic rewards (such as curiosity or entropy bonuses) to the exploration episodes to encourage information gathering. This provides a stronger exploration signal but requires hand-designing the intrinsic reward, which may not align with what is actually useful for task identification.
• Model-based exploration: MetaCURE (Zhang et al., 2020) trains a dynamics and reward prediction model $f(s', r \mid s, a, \mathcal{D}_{\text{tr}})$ and collects exploration data to make this model accurate. The intuition is that understanding the task dynamics is sufficient for task identification. However, this can be wasteful when the dynamics are complex but only a small aspect varies across tasks.

Summary of Meta-RL Approaches

The meta-RL landscape offers several design choices, each with characteristic strengths and weaknesses:

The choice between these paradigms depends on the specific problem characteristics. End-to-end methods are most appropriate when the exploration horizon is short and the reward signal is dense. Alternative strategies work well when standard exploration methods (like Thompson sampling) happen to align with the task structure. Decoupled methods shine when the exploration problem is complex but the task-relevant information is low-dimensional.

Looking Ahead

This lecture introduced meta-RL as an extension of multi-task RL where the agent must infer the task identity from interaction rather than receiving it as input. The black-box approach uses neural networks with memory to implement learned adaptation algorithms. The POMDP perspective reveals meta-RL as optimal decision-making under task uncertainty, inherently requiring a balance between exploration and exploitation.

Key takeaways:

• Meta-RL trains agents to rapidly adapt to new tasks from limited experience, learning a learning algorithm rather than a fixed policy.
• Black-box meta-RL uses recurrent or attention-based networks to accumulate task information and condition policy execution on the accumulated context.
• The POMDP perspective reveals that optimal meta-RL requires Bayes-optimal exploration—gathering information about the task while simultaneously maximizing reward.
• The coupling between exploration and execution is the central optimization challenge, motivating decoupled approaches.

In the next lecture, we examine the exploration problem in greater depth. We will study principled exploration strategies for bandits and MDPs—from upper confidence bounds to posterior sampling—and then return to the meta-learning setting to see how the DREAM algorithm elegantly decouples exploration from execution.