The Long-Horizon Challenge

Consider the everyday tasks that humans handle effortlessly: cooking a meal from scratch, driving across town, debugging a complex software system, or giving detailed feedback on a lengthy report. Each of these tasks involves hundreds or thousands of individual decisions, executed over extended time horizons. In the language of reinforcement learning, these are long-horizon tasks—problems where the agent must visit a very large number of states, make many sequential decisions, and navigate through a landscape full of pitfalls.

Three factors conspire to make long-horizon tasks especially challenging for standard RL and imitation learning methods:

1. Enormous state spaces. The number of states visited during a long task grows rapidly. A flat policy must learn to map every possible state to the right action, and the space of possible trajectories becomes intractably large.
2. Compounding errors. At every time step, there is an opportunity for the agent to make a mistake. Over hundreds of steps, even a small per-step error rate compounds into a high probability of overall failure. This is the same compounding-error phenomenon we encountered with behavioral cloning in earlier lectures.
3. Getting stuck. Without structure, exploration in a flat action space is unlikely to stumble upon the precise sequence of actions needed to complete a multi-stage task. The agent can easily enter states from which it cannot recover.

The Hierarchical Decomposition

The central idea of hierarchical reinforcement learning (HRL) is to decompose a difficult long-horizon task into a cascade of simpler sub-problems. Instead of a single monolithic policy that maps observations directly to low-level actions, we introduce at least two levels of decision-making:

The intermediate goals $\mathbf{g}_t$ go by many names in the literature: subgoals, subtasks, skills, options, or high-level actions. Regardless of terminology, they serve the same purpose—they provide a structured interface between the strategic, slow-thinking high level and the reactive, fast-acting low level.

Why Hierarchy Helps

Hierarchy is not merely an architectural convenience—it provides several concrete advantages over flat policies:

1. Supervision signal. The intermediate goals provide a supervision signal on how to complete the task, not just whether it was completed. The high-level policy decomposes the "what" while the low-level policy handles the "how."
2. Knowledge sharing across subtasks. A low-level skill like "pick up object" can be reused across many different high-level tasks (making a sandwich, cleaning a room, packing a box). This compositionality dramatically improves sample efficiency.
3. Structured exploration. In RL, the high-level policy explores in a much lower-dimensional goal space rather than the raw action space. Exploring over subgoals like "go to the kitchen" is far more efficient than exploring over individual joint torques.
4. Computational efficiency. In robotics, low-level controllers often must run at 50 Hz or higher for stable control, while high-level planning can operate at 1-5 Hz. A hierarchical architecture naturally accommodates these different time scales, running expensive reasoning models infrequently and cheap reactive controllers at high frequency.

Hierarchy vs. Flat Policies vs. Chain of Thought

It is worth asking: do we really need separate policies? There are three architectural paradigms to consider:

• Hierarchy: Two (or more) separate policies. $\pi_{\text{HL}}$ produces goals $\mathbf{g}_t$; $\pi_{\text{LL}}$ produces actions $\mathbf{a}_t$ conditioned on $\mathbf{g}_t$.
• Flat policy: A single policy $\pi$ maps directly from observations and the task prompt to actions $\mathbf{a}_t$.
• Chain of thought: A single policy $\pi$ that outputs both goals $\mathbf{g}_t$ and actions $\mathbf{a}_t$ together, akin to "thinking out loud" before acting.

The chain-of-thought approach can benefit from the same supervision signal as hierarchy. However, it may be too computationally expensive for real-time control (e.g., at 50 Hz) since the model must produce both the reasoning and the action at every step. As of now, there is no conclusive empirical comparison between these paradigms, and this remains an active area of research.

Design Choice 1: Goal Representations

The choice of how to represent intermediate goals $\mathbf{g}_t$ is perhaps the most critical design decision in hierarchical RL. The best representation will often be domain-specific, but good goal representations share three key properties:

1. Expressiveness. The representation must be able to communicate many different low-level behaviors. If the goal space is too restrictive, the high-level policy cannot express the full range of subtasks needed.
2. Structure. Similar behaviors should map to similar goals. A well-structured goal space enables generalization—if the low-level policy can accomplish goal $\mathbf{g}$, it should also be able to accomplish nearby goals $\mathbf{g}'$.
3. Appropriate level of abstraction. Goals should not be so abstract that the low-level policy cannot interpret them, nor so concrete that the high-level policy has to micromanage. The right abstraction level depends on the capabilities of both policies.

Common goal representations used in practice include:

• Language instructions (e.g., "pick up the bag", "move the scoop into the bowl"). Natural language is highly expressive, naturally structured via semantics, and easy for humans to provide as supervision. Language is particularly well-suited when the subtasks have clear semantic descriptions.
• Goal images (e.g., an image depicting the desired scene after the subtask). Images specify precisely what the world should look like, avoiding the ambiguity of language. They can be generated by image-editing models and can leverage large-scale unlabeled video data.
• State vectors (e.g., target positions of the robot and objects). State-based goals are precise and low-dimensional, making them efficient for RL training. However, they require access to ground-truth state information.
• Learned latent representations. The goal can live in a learned embedding space, but care must be taken to ensure the latent space is structured and interpretable—otherwise the hierarchy collapses to a flat policy.

Design Choice 2: Supervising Each Level

A hierarchical policy introduces a chicken-and-egg problem for training. The low-level policy and the high-level policy depend on each other:

• The low-level policy is trained to accomplish goals $\mathbf{g}$, not the original long-horizon task. But for which distribution of goals should it be trained? Ideally, whatever goals the high-level policy will output—but we have not learned a high-level policy yet.
• The high-level policy is trained to accomplish the original, long-horizon task by selecting appropriate subgoals. But it should be evaluated using the learned low-level policy—which we have not finished training either.

In practice, several strategies address this interdependency:

1. Sequential training. Train the low-level policy first on a broad distribution of goals (e.g., from demonstrations or self-play), then train the high-level policy using the frozen low-level policy. This is simple but the low-level may not cover all goals the high-level wants to issue.
2. Joint fine-tuning. After initial separate training, fine-tune both levels together so that each adapts to the other's behavior. At minimum, at least one policy should be adapted to the deficiencies of the other.
3. LLMs as high-level policies. Large language models can serve as effective high-level policies out of the box, since they already understand task decomposition through pre-training. This sidesteps the high-level training problem entirely.

Design Choice 3: When to Transition Between Subgoals

A critical question in hierarchical policy execution is: when should the high-level policy be re-queried to produce a new goal? There are two primary approaches, each with distinct trade-offs.

Option 1: Completion-Based Transitions

Re-query the high-level policy when the low-level policy has completed the current goal $\mathbf{g}_t$, estimated by measuring progress toward the goal (e.g., via a learned success classifier or distance metric).

• Advantage: Ideal in principle—each subgoal is fully executed before moving on.
• Disadvantage: Estimating when a subtask is "done" is itself a difficult learning problem. Errors in this estimation can be fatal: if the agent falsely believes it has completed a goal, it moves on prematurely; if it never detects completion, it gets stuck forever. These errors cause the agent to become perpetually stuck, making them more dangerous than errors in the alternative approach.

Option 2: Fixed-Interval Transitions

Re-query the high-level policy every $n$ time steps, regardless of whether the current goal has been completed.

• Advantage: Simple to implement and avoids the risk of permanent stalling.
• Disadvantage: Introduces a trade-off between computational cost and responsiveness. With small $n$, the high-level policy must reason at high frequency, increasing compute. With large $n$, the low-level policy may execute wrong actions for many steps if the high-level goal was poorly chosen. Additionally, small $n$ puts more burden on the high-level policy to make consistent, temporally-coherent plans.

In practice, the fixed-interval approach is more common because its failure mode (taking wrong actions for $n$ steps) is less catastrophic than the completion-based approach's failure mode (getting permanently stuck). Many systems use a relatively frequent replanning interval as a compromise.

Hierarchy in Modern Robot Learning Systems

Hierarchical architectures have become the dominant paradigm in state-of-the-art robot learning systems. Several prominent examples from 2025 illustrate how hierarchy is deployed at scale:

• Physical Intelligence $\pi_{0.5}$: Uses a pre-trained VLA (vision-language-action model) as a high-level policy that predicts language subtask descriptions (e.g., "pick up the pillow"), paired with an action expert that generates continuous robot actions. The high-level prompt provides the overall task ("clean the bedroom") and the system decomposes it into subtask predictions.
• NVIDIA Gr00t N1: Separates a vision-language model ("System 2" for high-level semantic reasoning) from a diffusion transformer ("System 1" for fast reactive control). The VLM processes language instructions and image observations to produce high-level plans, while the diffusion model generates motor actions at high frequency.
• Figure Helix: Employs a similar two-system architecture with an infrequent vision-language semantic reasoning module and a fast reactive control module that generates whole-body control at high frequency.
• Gemini Robotics: Uses a cloud-based backbone for high-level reasoning connected to a local action decoder for real-time control, explicitly separating the slow cloud inference from fast local execution.

Hierarchical Imitation Learning: Example Systems

Language Subgoals

One of the most natural instantiations of hierarchy uses language as the goal representation. This approach requires segmented demonstration data where each demonstration is divided into subtask segments, each annotated with a language description.

Data and Training

The training data consists of robot demonstrations segmented into sub-episodes, where each segment is labeled with a language command describing the subtask (e.g., "pick up the bag", "open the bag with scoop", "pour it into the bag"). Given this data, both levels of the hierarchy are trained with imitation learning:

• The high-level language policy takes RGB image observations and predicts the next language subtask command.
• The low-level LCBC (Language-Conditioned Behavioral Cloning) policy takes RGB images, joint states, and the language command, and predicts joint target actions.

Fine-Tuning with DAgger

A key advantage of the language-based hierarchy is that the high-level policy can be fine-tuned using only language supervision—no kinesthetic demonstrations required. The "Yell At Your Robot" system (Shi, Hu, Zhao, Sharma, Pertsch, Leo, Levine, Finn; RSS 2024) demonstrated this through hierarchical DAgger:

1. Deploy the hierarchical policy with the low-level policy frozen.
2. A human observer provides language corrections when the high-level policy makes a wrong subtask prediction (e.g., saying "avoid pouring outside the bag" when the robot is about to spill).
3. These language corrections override the high-level policy's prediction and are recorded as training data.
4. The high-level policy is updated via gradient descent on the collected corrections, following the DAgger paradigm.

After fine-tuning, the high-level policy internalizes the corrections and can self-correct—it learns to predict the right subtask without human intervention. Experimental results from Hi Robot (Shi et al., ICML 2025) show that hierarchy provides a 19-34% improvement over flat VLA policies on long-horizon tasks, and DAgger-based fine-tuning of the high-level policy closes an additional 20% gap toward oracle (human-corrected) performance.

Image Subgoals

An alternative to language goals is to use images as the intermediate representation. In this paradigm, the high-level policy produces a goal image depicting what the scene should look like after completing the next subtask, and the low-level policy is conditioned on this goal image rather than a language command.

SuSIE: Subgoal Synthesis via Image Editing

The SuSIE system (Black, Nakamoto, Atreya, Walke, Finn, Kumar, Levine; ICLR 2024) implements this idea by using an image editing model as the high-level policy. Given the current observation and a language task description, the image editor generates a modified version of the current image showing the scene after the next subtask is completed. The low-level policy is a goal-image-conditioned controller trained to reach the depicted scene.

This approach offers several advantages over language-based hierarchy:

• No segmented language annotations needed. The high-level policy can be trained on raw video data without per-segment language labels, significantly reducing the annotation burden.
• Unlabeled video data. The high-level image-editing model can incorporate large amounts of unlabeled video data—including videos of humans performing tasks—to improve its understanding of what intermediate states should look like.
• Unambiguous specification. Images avoid the ambiguity inherent in natural language by specifying precisely what the world should look like.

Experiments demonstrate that including human videos in the high-level policy training data improves performance on manipulation tasks. For example, on a suite of tabletop tasks, adding the Something-Something human video dataset to robot-only data (BridgeData) improved average success rates from 0.30 to 0.50 on harder scenes and from 0.80 to 0.88 on easier scenes.

Hierarchical Reinforcement Learning

While the imitation-based systems described above learn hierarchical policies from demonstrations, hierarchical reinforcement learning addresses the setting where the agent must discover both the goal decomposition and the low-level skills through trial and error.

HIRO: State-Based Goal-Conditioned Hierarchy

The HIRO system (Nachum, Gu, Lee, Levine; NeurIPS 2018) represents goals as state vectors—specifically, relative target positions of the agent and objects. The architecture works as follows:

• The low-level policy is a goal-conditioned policy trained with a simple goal-reaching reward: it receives reward proportional to how close the current state is to the target goal state.
• The high-level policy is trained to output goal states (which serve as the high-level "actions"). It is optimized to maximize the task reward using an off-policy RL algorithm.
• Hindsight relabeling is used on the high-level actions: after a trajectory is collected, the high-level actions are relabeled with the goals that were actually achieved, improving sample efficiency.

Language-Conditioned Hierarchical RL

Jiang, Gu, Murphy, and Finn (NeurIPS 2019) extended hierarchical RL to use language as the goal space. In this approach:

• The low-level policy is language-conditioned and uses hindsight language relabeling—after a trajectory segment, a language model generates a description of what actually happened, which is used to relabel the training data.
• The high-level policy is trained to output language strings that maximize the task reward, with the language serving as an abstraction for directing low-level behavior.

Using language as the goal space in RL is particularly appealing because language inherits the structure of natural language semantics, making the goal space naturally well-organized. However, it requires a mechanism for grounding language in the environment, which can be challenging without pre-training.

Unsupervised Skill Discovery

A related research direction asks: can we discover a set of diverse skills without any supervision? The DIAYN framework (Eysenbach, Gupta, Ibarz, Levine; ICLR 2019) addresses this by training a set of skills that are maximally distinguishable from each other, using a mutual information objective. The learned skills can then serve as the action space for a high-level policy, providing a form of unsupervised hierarchy.

Summary and Open Questions

Hierarchical reinforcement learning addresses the fundamental challenge of long-horizon decision-making by decomposing difficult tasks into manageable subtasks. The key design decisions are:

1. Goal representation: Language, images, state vectors, or learned embeddings—each with different trade-offs in expressiveness, structure, and supervision requirements.
2. Supervision for each level: The high-level and low-level policies can be trained separately or jointly, via imitation learning, RL, or a combination. The co-dependence between levels requires careful handling.
3. Subgoal transition timing: Completion-based transitions are ideal but fragile; fixed-interval transitions are simpler and more robust in practice.

Several important open questions remain in this area:

• Fine-tuning hierarchical systems with RL. Most large-scale hierarchical robot learning systems are trained with imitation learning. Fine-tuning them with RL to improve beyond the demonstration data remains an important and largely open research direction.
• Hierarchy vs. chain-of-thought. Whether explicit hierarchy (separate models) outperforms implicit hierarchy (a single model that reasons before acting) is an open empirical question, especially as foundation models grow more capable.
• Automatic subtask discovery. Current systems largely rely on human-defined task decompositions. Learning to automatically discover the right level of abstraction and the right set of subtasks from data is a long-standing challenge.