Reinforcement Learning for Robotics

This guest lecture by Ashish Kumar (UC Berkeley) focuses on the application of reinforcement learning to robotics, with particular emphasis on the problem of transferring policies trained in simulation to real-world robots. The central thesis is that RL has achieved remarkable successes—from AlphaGo to reasoning in large language models—and the key enablers in every case are the same: well-specified rewards and scalable evaluation of policies.

Before diving into the robotics-specific content, it is useful to distinguish between two paradigms for learning behavior:

• Imitation learning — used in vision-language models, video generation, and image segmentation. It relies heavily on off-policy data, learns from good examples, and tends to produce generalist policies.
• Reinforcement learning — used in game playing, reasoning, and robotics. It relies on on-policy data, learns from both good and bad experience, and tends to produce specialist policies optimized for specific tasks.

Across the major RL success stories, a common pattern emerges. AlphaGo learned to exceed human capability in Go. DeepSeek-R1 used RL to incentivize coherent long-horizon reasoning in LLMs. And in robotics, RL has enabled unprecedented reliability and dexterity. In each case, the breakthrough required two ingredients: a reward signal that could be evaluated programmatically (or at scale), and a way to evaluate millions of policy rollouts cheaply.

The Simulation-to-Reality Gap

Simulation provides a compelling training ground for robot learning. Modern physics engines like IsaacGym can simulate thousands of robots in parallel on a single GPU, generating the massive amounts of on-policy experience that RL algorithms demand. Rewards can be computed from simulator state variables that would be difficult or impossible to measure on a physical robot—exact contact forces, ground-truth positions of every link, and precise terrain geometry.

However, simulation is an approximation of reality. The parameters of the simulated environment—friction coefficients, motor characteristics, masses, moments of inertia, terrain profiles—will inevitably differ from those of the real world. A policy that is optimal in simulation may fail catastrophically on the physical robot because it has learned to exploit dynamics that do not exist in reality, or has never encountered dynamics that do.

There have been three main approaches to bridging this gap, each with its own limitations:

• Domain randomization — train a single robust policy across a wide distribution of simulator parameters. The hope is that if the policy works across many simulated environments, the real world will be "just another sample" from that distribution. This produces a conservative policy that may sacrifice performance for robustness.
• System identification — carefully measure the physical parameters of the real robot and calibrate the simulator to match. This can work well for rigid-body dynamics but becomes intractable for complex, changing environments (mud, ice, deformable objects).
• Fine-tuning at test time — deploy a pre-trained policy on the real robot and continue training with real-world data. This requires additional real-world interaction, which is expensive and potentially dangerous.

The key contribution of this lecture is a fourth approach: rapid motor adaptation, which trains a policy that can infer the current environment parameters from its own sensory experience and adapt its behavior online, without any explicit system identification or fine-tuning.

Rapid Motor Adaptation (RMA)

Rapid Motor Adaptation (RMA), developed by Kumar, Fu, Pathak, and Malik, introduces an architecture that enables a single policy to walk across diverse terrains and conditions by rapidly adapting to changes in the environment. The core idea is to decompose the problem into two parts: a base policy that maps the current state and an environment encoding to actions, and an adaptation module that estimates the environment encoding from recent sensory history.

The RMA Architecture

During training in simulation, the environment is characterized by a set of extrinsic parameters $e$ that are known to the simulator but would be unavailable on the real robot:

• Mass and center of mass of the robot body
• Friction coefficients of the ground surface
• Terrain height profile
• Motor strength and damping characteristics

The architecture consists of two neural network modules:

1. Environmental Factor Encoder: A network that maps the extrinsic parameters $e$ to a compact latent vector $z_t \in \R^k$, capturing the aspects of the environment most relevant to locomotion.
2. Base Policy: A network $\pi(a_t \mid x_t, a_{t-1}, z_t)$ that takes the current proprioceptive observation $x_t$ (joint positions, joint velocities, body orientation), the previous action $a_{t-1}$, and the environment encoding $z_t$, and outputs the action $a_t$ (target joint positions).

Reward Function Design

The reward function for locomotion training is carefully designed to produce natural, energy-efficient, and hardware-safe gaits. It combines multiple terms organized into three categories:

• Forward walking: Rewards for tracking a commanded forward velocity, penalizing deviation from the target speed.
• Energetics: Penalties on joint torques and joint power consumption, encouraging the policy to find energy-efficient gaits. These terms are inspired by biomechanics research on animal locomotion.
• Stability and hardware protection: Penalties on ground impact forces, excessive body roll and pitch, and sudden changes in action. These terms protect the physical hardware from damage and encourage smooth, stable gaits.

Two-Phase Training

RMA uses a two-phase training procedure, both conducted entirely in simulation. The key insight is that the adaptation module can be trained after the base policy, using the base policy's own experience as supervision.

Phase 1: Training with Privileged Information

In the first phase, both the environmental factor encoder $\mu$ and the base policy $\pi$ are trained jointly using PPO. The environment parameters $e$ are randomized at the start of each episode (domain randomization over masses, frictions, terrain types, motor characteristics), and the encoder has direct access to these parameters. This is privileged information—data that is available during training but not during deployment.

The resulting policy learns to condition its behavior on the environment encoding $z_t = \mu(e)$, adapting its gait to different surfaces, payloads, and terrain profiles. Because the encoder compresses the high-dimensional extrinsic parameters into a compact latent vector, the base policy effectively learns a family of behaviors parameterized by $z_t$.

Phase 2: Learning to Estimate Extrinsics from History

In the second phase, the base policy is frozen, and a new adaptation module $\hat{\mu}$ is trained to predict the environment encoding $z_t$ from a window of recent proprioceptive observations and actions:

where $H$ is the history length (typically 50 timesteps). The adaptation module is trained to regress against the output of the privileged encoder: $\hat{z}_t \approx z_t = \mu(e)$. This is a supervised learning problem, with training data generated by rolling out the frozen base policy across randomized environments.

Deployment and Real-World Adaptation

At deployment time, the environmental factor encoder is discarded entirely. Only the adaptation module and the base policy are deployed on the robot. The adaptation module runs at 10 Hz, consuming a sliding window of the last 50 proprioceptive observations and actions to produce the environment encoding $\hat{z}_t$. The base policy runs at 100 Hz, taking the most recent proprioceptive observation, the previous action, and the latest $\hat{z}_t$ to compute motor commands.

This architecture enables continuous online adaptation. As the robot transitions from one surface to another, or as an external perturbation is applied, the adaptation module detects the change in dynamics through the observation history and adjusts $\hat{z}_t$ accordingly. The base policy then modifies its gait to match the new conditions, all without any explicit system identification or parameter estimation.

Indoor Evaluation

The RMA system was evaluated on a Unitree A1 quadruped robot in a variety of challenging indoor conditions designed to stress-test the adaptation mechanism:

• Oily surfaces with plastic-wrapped feet: The robot's feet were wrapped in plastic and placed on an oily surface, drastically reducing friction. The adaptation module detected the low-friction condition and adjusted the gait to maintain stability.
• 5 kg payload throw: While the robot was walking, a 5 kg payload (roughly 40% of the robot's body weight) was thrown onto its back. The adaptation module detected the sudden mass change and adjusted the joint torques and gait timing to compensate.
• Uneven planks: The robot walked across loose planks that shifted and tilted underfoot, creating constantly changing terrain. The adaptation module continuously updated its terrain estimate.

Analysis of the Adaptation Module

Inspection of the learned latent vectors reveals that the adaptation module captures physically meaningful quantities. When the robot encounters an oily surface, specific dimensions of $\hat{z}_t$ shift to values that correlate with reduced friction, and the corresponding knee torques increase to compensate. When a payload is thrown onto the robot, different latent dimensions respond, reflecting the mass change, and the torques across all legs increase to support the additional weight.

Quantitative comparisons show that RMA significantly outperforms both domain randomization (which produces overly conservative gaits) and policies without adaptation (which fail when the environment deviates from the training distribution). The adaptation mechanism provides the crucial ability to specialize behavior to the current conditions while maintaining the robustness provided by training across diverse environments.

Vision-Based Locomotion

Proprioceptive adaptation enables robust locomotion on flat and gently varying terrain, but for truly challenging environments—stairs, stepping stones, gaps, discrete obstacles—the robot needs to see where it is going. The question becomes: how should we incorporate vision into the locomotion policy?

The Problem with Terrain Maps

The conventional approach in robotics is to build an explicit terrain map from sensor data (typically LiDAR or depth cameras), then plan footstep placements on this map. Systems like those described by Miki et al. (Science Robotics, 2022) and Kim et al. (ICRA, 2020) construct elevation maps of the surrounding terrain and feed them to a locomotion controller.

However, terrain maps constructed from real-world sensor data are extremely noisy. Depth sensors produce artifacts near edges, transparent surfaces, and in outdoor lighting conditions. The noise in the map can overwhelm the signal that the controller needs to make good decisions. As terrain difficulty increases, the signal-to-noise ratio deteriorates further, and map-based controllers can perform worse than blind controllers that rely solely on proprioception.

Egocentric Depth: Tightly Coupling Vision and Control

The egocentric depth approach, developed by Agarwal and Kumar et al., replaces terrain maps with raw egocentric depth images from a forward-facing camera mounted on the robot. The depth image is fed directly into the locomotion policy, which learns to extract the relevant terrain information end-to-end. This approach follows the same two-phase privileged learning paradigm as RMA.

Phase 1: Training with Privileged Terrain Information

In the first phase, the policy is trained in simulation with access to privileged terrain information in the form of scandots—a dense grid of height measurements sampled at the robot's foot positions. The policy also has access to privileged extrinsic parameters (friction, payload). Two RNN modules encode these privileged inputs: one produces a terrain encoding $\gamma_t$ from the scandots $m_t$, and the other produces an environment encoding $z_t$ from the extrinsics $e_t$. A base policy MLP then maps $(o_t, \gamma_t, z_t)$ to actions, where $o_t$ is the proprioceptive observation.

Training uses PPO in IsaacGym across a rich curriculum of terrain types: stairs, slopes, rough flat ground, gaps, stepping stones, and discrete obstacles. The reward combines velocity tracking and energy minimization terms.

Phase 2: Distilling Vision from Privileged Information

In the second phase, the Phase 1 policy is frozen, and a new student network is trained to produce the same terrain encoding $\hat{\gamma}_t$ and environment encoding $\hat{z}_t$ from egocentric depth images $d_t$ and proprioceptive history $o_{t-H:t}$. A depth encoder processes the egocentric depth image, and two RNN modules produce the estimated encodings. The student is trained by regression against the teacher's encodings.

The resulting deployment policy uses only egocentric depth and proprioception—both readily available on a physical robot. No terrain map is constructed at any stage.

Results and Emergent Behaviors

The egocentric depth locomotion system was evaluated on an A1 quadruped navigating environments that are particularly challenging for small robots. Standard staircases, for example, have a 17 cm rise and 26 cm run—dimensions designed for human legs, not for a small quadruped whose leg length is comparable to the stair height.

Quantitative Performance

The system was compared against two baselines across four terrain types, measuring average distance traversed before failure:

Two patterns stand out. First, the egocentric depth approach dominates across all terrain types. Second, the performance gap grows dramatically as terrain difficulty increases. On stepping stones, the egocentric method travels 18 times further than either baseline. On stairs, the map-based method actually performs worse than blind locomotion—noisy depth maps near stair edges actively mislead the controller.

Emergent Behaviors

One of the most striking findings is the emergence of sophisticated locomotion strategies that were never explicitly programmed. Because no predefined gait is imposed, the policy discovers several remarkable behaviors:

• Hip abduction on stairs: When climbing stairs whose dimensions exceed the robot's nominal stride length, the policy learns to splay its legs outward (hip abduction), effectively widening its base of support. This closely resembles strategies used by animals navigating challenging terrain.
• Emergent footstep planning: The policy implicitly plans footstep placements multiple steps ahead, positioning its feet to ensure stability through sequences of obstacles rather than greedily optimizing each individual step.
• Gait-free locomotion: Rather than using a fixed gait pattern (trot, gallop, etc.), the policy dynamically adjusts its gait in response to terrain—walking cautiously on uneven ground, trotting on flat sections, and using entirely novel footfall patterns on stairs.

Beyond Locomotion: Dexterous Manipulation and Flight

The privileged-learning and rapid-adaptation paradigm is not limited to legged locomotion. The same principles have been successfully applied to other domains in robotics.

In-Hand Dexterous Manipulation

Qi et al. (CoRL 2022) applied rapid motor adaptation to in-hand object rotation using a multi-fingered robot hand. The task is to continuously rotate an object held in the hand—a dexterity benchmark that requires precise contact management.

The system generalizes across objects with dramatically different physical properties:

• Weight: Objects ranging from 5 g to 198 g
• Coefficient of friction: From very slippery to highly grippy surfaces
• Center of mass: Both above and below the finger contact points
• Shape: Cubes, cylinders, and even cups

The adaptation module uses proprioceptive history for implicit contact detection. When a finger is in contact with the object, the measured joint position deviates from the commanded action (because the object resists the finger's motion). When there is no contact, the joint position closely tracks the commanded action. By observing these discrepancies across the history window, the adaptation module infers which fingers are currently in contact and adjusts the manipulation strategy accordingly—all without any explicit tactile sensors.

Agile Drone Flight

The same rapid adaptation framework has also been applied to drone flight, where the environment parameters that must be adapted to include wind conditions, payload mass, and aerodynamic effects. The drone's adaptation module estimates these factors from the discrepancy between commanded thrust/torque and observed accelerations, enabling stable flight even under conditions not seen during simulation training.

Open Challenges and Future Directions

Despite the impressive results demonstrated in this lecture, several fundamental challenges remain in RL for robotics.

Humanoid Robots

Extending these techniques to humanoid robots is an active frontier. Humanoids have far more degrees of freedom than quadrupeds, making the locomotion problem significantly harder. The balance requirements are more stringent (bipedal balance on two feet versus quadrupedal stability on four), and the diversity of tasks a humanoid must perform—walking, climbing, manipulating objects, navigating human environments—is much broader.

Simulation Technology

Current simulation technology, while powerful, remains primitive in important respects. Many real-world phenomena—deformable objects, fluids, granular media, soft contacts—can be modeled in simulation but at prohibitive computational cost. Rich Sutton's "bitter lesson" suggests that the answer is to bet on compute: general methods that leverage increasing computational resources tend to dominate hand-engineered approaches over the long run. The implication for robotics simulation is that investing in more powerful and general-purpose simulators will eventually pay larger dividends than clever engineering of specific simulation shortcuts.

General Reward Models

The reward functions in the systems described in this lecture were hand-designed for specific tasks. A major open question is whether large pre-trained models (foundation models trained for general understanding) can serve as general-purpose reward functions for robotics. This is appealing because such models encode rich world knowledge, but RL will ruthlessly exploit any weaknesses in the reward model—a phenomenon known as reward hacking. As a worst case, if automated reward models prove unreliable, human labeling of robot behavior remains a viable fallback, though an expensive one.

Superhuman Capabilities

Perhaps the most exciting long-term direction is achieving superhuman robotic capabilities. The lesson from AlphaGo is instructive: a combination of sparse reward and extensive search enabled the discovery of strategies that no human had ever conceived. Can the same approach yield robots that exceed human capability in physical tasks? Achieving this would require scalable search (or exploration) in the space of physical behaviors—a much harder problem than search in a board game, but one where the potential impact is enormous.

Summary

This lecture presented a principled approach to the sim-to-real transfer problem in robot learning, organized around the idea of rapid motor adaptation. The key takeaways are:

• The two ingredients for RL success in robotics are well-specified rewards (computable from simulator state) and scalable policy evaluation (through massively parallel physics simulation).
• Rapid Motor Adaptation (RMA) bridges the sim-to-real gap by training a base policy with privileged environment information, then distilling an adaptation module that estimates the same information from proprioceptive history. The adaptation module runs online, enabling the robot to continuously adjust to changing conditions.
• Egocentric depth locomotion extends this paradigm to vision, replacing noisy terrain maps with end-to-end learned depth processing. The resulting system is both map-free and gait-free, discovering emergent locomotion strategies that outperform engineered baselines.
• The privileged learning paradigm—train with information available only in simulation, then distill into a deployable architecture—is broadly applicable beyond locomotion, with demonstrated success in dexterous manipulation and drone flight.
• Open challenges include scaling to humanoid robots, improving simulation fidelity, developing general reward models, and ultimately achieving superhuman physical capabilities.

In the final lecture, we will step back to survey the broader landscape of open problems and future directions in deep reinforcement learning, covering challenges in reward specification, scaling, safety, evaluation, and the practice of RL research itself.