Why Aren't Robots Autonomous Already?

Reinforcement learning is, at its core, a trial-and-error process. The agent executes actions from a policy $\pi$, observes data from the environment, updates the policy, and repeats. In simulation, this loop is effortless—the environment can be reset to any initial state with a single function call (env.reset()). But in the real world, there is no reset button.

Consider a robot learning to shelf a book. Through trial and error, the robot may attempt the task thousands of times before finding a reliable strategy. After each attempt—whether the book lands on the shelf or falls to the floor—someone must walk over, pick up the book, reposition the robot's arm, and set up the scene again. This human supervision at every episode boundary is one of the primary bottlenecks preventing RL from scaling on real robots.

The same problem arises across domains: a robot learning to navigate must be carried back to a starting position after each trial; a manipulation robot must have its workspace tidied by a person; a legged robot that falls over must be placed upright again. Each of these resets requires human intervention, making the data collection process fundamentally non-autonomous.

Defining Autonomous RL

What if we simply increased the episode length $H$ so that the robot interacts with the world for a very long time between resets? Fewer resets mean less human supervision. Taken to the extreme, we arrive at the definition of autonomous RL:

Two Evaluation Criteria

In autonomous RL, we may care about two different evaluation criteria, depending on the application:

1. Deployed Policy Evaluation: Measures the quality of the learned policy when evaluated from the initial state distribution $\rho_0$ after training is complete. This is the standard RL objective: $$J(\pi) = \E_{s_1 \sim p(s_1), a_t \sim \pi}\left[\sum_t \gamma^t r(s_t, a_t)\right]$$ This is appropriate when the robot is being trained to perform a task that will later be deployed (e.g., a robot chef that trains in the kitchen and then must reliably cook meals).
2. Continuing Policy Evaluation: Measures the total reward accumulated over the agent's entire lifetime: $$\lim_{h \to \infty} \E\left[\frac{1}{h}\sum_{t=0}^{h} r(s_t, a_t)\right]$$ This is appropriate when the agent must perform well during training, not just after (e.g., a Mars rover that must collect useful data from day one, since there is no "deployment" phase separate from its operational life).

Why Autonomous RL Matters

The connection between robotics and autonomy runs deep. We want robots not only to operate with minimal human supervision, but also to train with minimal human supervision. RL on real robots is a promising path toward high success rates and reliability—but only if data collection can scale. Autonomy unlocks more data, and more data may unlock greater generalization. This is the fundamental motivation for the algorithms we study in this lecture.

The Challenges of Learning Autonomously

What happens when we simply apply standard RL algorithms (like SAC) to the autonomous setting by increasing the episode length? Empirical results reveal a stark degradation: as the episode length increases from 1,000 to 100,000 steps, the quality of the deployed policy drops dramatically. With episodes of 100,000 steps, the agent barely learns at all.

Two fundamental challenges explain this failure:

Challenge 1: Exploration Drift

In episodic RL, exploration that leads the agent far from the goal is harmless—the next episode resets the agent to a good starting state. In the autonomous setting, there is no such safety net. Exploratory actions can cause the agent to drift far away from the region of state space where it can productively practice the task. A robot trying to learn to place a cup in a coffee machine might accidentally knock the cup off the table; without a reset, it is now in a state from which it cannot practice the original task at all.

Challenge 2: State Distribution Collapse

As the agent drifts further from useful states, its experience becomes concentrated in a narrow, uninformative region of the state space. The agent's state distribution collapses to states that are far from both the initial state distribution and the goal. Since the agent never returns to states from which it could practice the task, it never learns a good policy. This creates a vicious cycle: a poor policy leads to bad states, which leads to uninformative data, which leads to an even worse policy.

Forward-Backward RL

The key observation is that standard RL succeeds when the agent can frequently retry from the initial state distribution. If we cannot get manual resets, can we instead learn a policy to reset? This is the central idea behind Forward-Backward RL (FBRL).

The forward policy $\pi_f$ tries to accomplish the task, while the backward policy $\pi_b$ acts as a learned reset controller that returns the environment to a state near $\rho_0$ so the forward policy can try again. By alternating between forward and backward phases, the agent autonomously practices the task without human intervention.

FBRL is simple and effective, but has a notable limitation: the backward policy is ultimately discarded at test time. All the compute and data spent training $\pi_b$ serve only to facilitate training of $\pi_f$—the backward policy itself has no value at deployment.

MEDAL: Matching Expert Distributions for Autonomous Learning

A key improvement over vanilla FBRL is to modify the backward policy's reward function. Rather than simply trying to reach a fixed initial state, the backward policy in MEDAL learns to match the state distribution found in expert demonstrations.

MEDAL addresses both challenges of autonomous RL simultaneously:

• Challenge 1 (exploration drift): The backward policy steers the agent back toward states that resemble the demonstrations, preventing it from drifting to uninformative regions.
• Challenge 2 (state distribution collapse): By resetting to a distribution that matches the demonstrations, the backward policy ensures that the forward policy encounters a diverse set of initial states—both easy and hard—improving sample efficiency. This connects to the theoretical insight of Kakade and Langford (ICML 2002) that training from a wide distribution of start states leads to better policies.

Experimental Results on the EARL Benchmark

The EARL benchmark (Sharma, Xu et al.; ICLR 2022) provides standardized evaluation for autonomous RL algorithms. Training involves very infrequent resets (every 200k steps), and evaluation measures the deployed policy's performance from $\rho_0$. Results on tasks like tabletop manipulation and door closing show a clear hierarchy of methods:

• MEDAL achieves the best overall performance, often matching or approaching the oracle SAC baseline (which gets free episodic resets).
• VaPRL (goal curriculum) and FBRL (reach $\rho_0$) are competitive but slower to converge.
• R3L (state novelty-based reset) shows moderate performance.
• Non-episodic SAC (no backward policy at all) essentially fails to learn, confirming that standard RL algorithms cannot handle the autonomous setting.

Forward-backward approaches have also been demonstrated on real robots. Vanilla FBRL has been applied with demo-initialized policies and VLM-provided rewards, while MEDAL has been used with learned rewards, both achieving autonomous training on physical manipulation tasks.

Autonomous RL with Task Cycles

An alternative to the forward-backward paradigm is to structure the training process as a cycle of related tasks that naturally reset each other. Instead of learning a separate backward policy, the agent practices multiple tasks in sequence, where completing (or failing) one task naturally sets up the initial conditions for another.

This approach has been demonstrated in several real-world settings:

• Dexterous manipulation (Gupta, Yu, Zhao, Kumar, Rovinsky, Xu, Devlin, Levine; 2021): A robot hand learns to re-center, flip, lift, and rotate objects in a cycle, with each manipulation naturally setting up the next.
• Legged locomotion (Smith, Kew, Peng, Ha, Tan, Levine; 2021): A quadruped robot fine-tunes locomotion skills in the real world by cycling through forward and backward walking, turns, and other gaits.
• Multi-step manipulation (Han, Levine, Abbeel): A PR2 robot learns a 7-stage wrench task by cycling through the stages, where the forward execution of each stage naturally leads into the next.

Task Proposal: An Open Research Question

A critical question in the task-cycle approach is: how does the agent decide which task to practice next? Given its current state, the agent must identify which task is both (a) feasible from the current state and (b) useful for improving its overall competence.

This is generally an open research question. One promising approach, explored in SOAR (Zhou, Atreya, Lee, Walke, Mees, Levine), is to ask a vision-language model. Given the current observation, a VLM can identify what tasks are possible in the current state and suggest which one the robot should practice. This leverages the VLM's broad world knowledge to guide the autonomous training process.

Forward-Forward RL

A third alternative is to design the task setup so that it can be done repetitively without any explicit reset. For instance, if a robot's task is to push an object back and forth between two positions, completing the forward push naturally sets up the backward push, and vice versa. This "forward-forward" approach avoids the need for either a backward policy or a task cycle, but requires careful environment design to ensure the task is inherently cyclical.

Single-Life Reinforcement Learning

The autonomous RL methods discussed so far focus on the training phase: how to collect data without human resets. But what happens after training, when the policy is deployed in the real world?

In a controlled training environment, things go roughly as planned. But the real world is complex, and the deployment environment will inevitably differ from the training environment. The robot may encounter novel objects, unexpected obstacles, or different lighting conditions. Even humans make mistakes and must adapt on the fly—consider the mundane experience of approaching a door, finding it locked, and adjusting your strategy to use a key. Can robots do the same?

SLRL differs from both episodic RL and reset-free RL in a fundamental way:

• Episodic RL: Multiple episodes in both training and testing, with resets between episodes.
• Reset-free RL: A single long episode during training (no resets), but the training and test environments are the same.
• Single-life RL: Prior training data is available, but the agent has only one episode in a novel test environment. It must adapt on the fly.

Why Vanilla RL Fails at Test Time

A natural approach is to simply run fine-tuning with RL at test time, using the test-environment reward signal. However, this typically fails: the agent encounters out-of-distribution states early in the episode, makes errors that compound, and drifts into irrecoverable states before the RL algorithm can meaningfully update the policy. Without resets, a single mistake can doom the entire episode.

Approaches to Single-Life RL

Several ideas can help make single-life RL feasible:

1. Guiding the agent toward familiar states. Rather than relying on unconstrained exploration, add an auxiliary objective that steers the agent toward states that resemble those in the prior training data (Chen, Sharma, Levine, Finn; NeurIPS 2022). This prevents the catastrophic drift that occurs with vanilla test-time RL.
2. Adaptation in skill space, not action space. If the agent has pre-trained skills (e.g., from hierarchical training), it can adapt at the level of skill selection rather than individual actions. This dramatically reduces the search space for adaptation. For example, RMA (Kumar, Fu, Pathak, Malik; RSS 2021) enables legged robots to rapidly adapt their locomotion by adjusting a small number of latent parameters rather than relearning from scratch.
3. Leveraging foundation models for reasoning. Pre-trained LLMs or VLMs can reason about how to adapt behavior in novel scenarios without requiring extensive environment interaction. For instance, Chen, Lessing, Tang, Chada, Smith, Levine, and Finn (ICRA 2025) use vision-language models to provide commonsense reasoning for legged robot adaptation, enabling the robot to identify and respond to environmental changes that it has never encountered during training.

Summary

This lecture examined the unique challenges of applying reinforcement learning to physical robots that must operate and learn in the real world. The key ideas are:

1. The reset problem. Standard RL assumes free environment resets between episodes. On real robots, resets require human supervision, creating a fundamental bottleneck.
2. Autonomous RL. By eliminating or minimizing resets, autonomous RL enables robots to train with minimal human intervention. However, this introduces two new challenges: exploration drift and state distribution collapse.
3. Forward-backward RL. Training a backward policy to return the agent to good starting states is a simple and effective approach. MEDAL improves on vanilla FBRL by learning a reward function that matches the backward policy's state distribution to that of expert demonstrations.
4. Task cycles. Structuring training as a cycle of related tasks that naturally reset each other provides an elegant alternative to explicit backward policies. Choosing which task to practice next remains an open research question, with VLMs offering a promising direction.
5. Single-life RL. After training, the agent must adapt to novel environments in a single episode. Guiding exploration toward familiar states, adapting in skill space rather than action space, and leveraging foundation models for reasoning are key strategies for this challenging setting.

The overarching goal—building autonomous agents that can learn and interact in the real world—remains one of the most important and challenging problems in deep RL. The algorithms covered in this lecture represent the current frontier, but many open problems remain, particularly around scaling these methods to diverse, unstructured real-world environments.