What Is Deep Reinforcement Learning?

Deep reinforcement learning sits at the intersection of two powerful ideas: reinforcement learning, the study of how agents learn to make sequential decisions from experience, and deep learning, the use of deep neural networks as flexible function approximators. Together, they provide a framework for tackling problems where a system must make multiple decisions over time based on a stream of information—observing, taking an action, observing again, taking another action, and so on.

More precisely, CS 224R studies both sequential decision-making problems and the solutions to those problems. The course covers a broad set of methods:

• Imitation learning — learning behavior by mimicking expert demonstrations
• Model-free and model-based RL — learning from reward signals, with or without a learned dynamics model
• Offline and online RL — learning from pre-collected datasets versus learning through active interaction
• Multi-task and meta-RL — generalizing behavior across tasks and learning to learn
• RL for LLMs — using reinforcement learning to fine-tune large language models
• RL for robots — applying deep RL to physical manipulation and locomotion

The emphasis throughout is on solutions that scale to deep neural networks—methods that can handle high-dimensional observations like images, complex continuous action spaces, and the kinds of large-scale training pipelines used in modern AI systems.

How Does Deep RL Differ from Supervised Learning?

In standard supervised learning, we are given a dataset of labeled input-output pairs $\{(x_i, y_i)\}$ and learn a function $f(x) \approx y$. Two key properties make this setting tractable: (1) the learner is directly told what to output for each input, and (2) the data points are typically assumed to be independently and identically distributed (i.i.d.).

In reinforcement learning, neither of these properties holds. Instead of learning a mapping from inputs to labels, we learn a behavior $\pi(a \mid s)$—a policy that maps states (or observations) to actions. The key differences are:

• Indirect feedback: The agent is not told which action is correct. Instead, it receives a scalar reward signal after acting, and must figure out which actions led to good outcomes.
• Non-i.i.d. data: The agent's actions $a$ affect future observations. The data the agent sees depends on the decisions it has already made, creating complex dependencies that violate the i.i.d. assumption.

The scope of "behavior" in RL is remarkably broad. It encompasses motor control for robots, dialogue generation for chatbots, strategic play in complex games, driving decisions for autonomous vehicles, and action sequences for web agents—essentially any setting where an intelligent system must make a sequence of choices.

Why Study Deep Reinforcement Learning?

There are four compelling reasons to study deep RL, ranging from practical deployment to fundamental scientific questions.

1. Going Beyond Supervised $(x, y)$ Examples

Decision-making problems are everywhere, and they do not fit neatly into the supervised learning paradigm. Several situations demand an RL perspective:

• AI agents of all kinds: Robots, autonomous vehicles, and web assistants must make sequential decisions in dynamic environments.
• Systems that interact with people: Chatbots and recommender systems operate in a loop with users, where each response shapes the conversation or the user's future behavior.
• Feedback loops: When deploying an AI system affects future outcomes and observations—for example, a content recommendation algorithm that shapes what users want to see next—supervised learning's static assumptions break down.
• Non-differentiable or non-standard objectives: When you lack labels entirely, or your objective is not simple prediction accuracy (and may not even be differentiable), RL lets you learn from any objective that can be expressed as a reward signal.

2. Widely Used and Deployed for Performant AI Systems

Deep RL is not a purely academic pursuit—it powers some of the most impressive AI systems in production today. The range of successful applications is striking:

• Legged robotics: Companies like Unitree use RL-trained policies for both quadruped and humanoid locomotion, enabling robots to walk, run, and navigate complex terrain.
• Robot manipulation: Physical Intelligence's $\pi_0$ system and Google's Gemini Robotics use deep RL to learn dexterous manipulation tasks—folding laundry, clearing tables, and handling novel objects.
• Complex games: DeepMind's AlphaGo achieved superhuman performance in Go, with its famous "Move 37" in the match against Lee Sedol surprising expert commentators—a move no human would have considered, showcasing RL's ability to discover new solutions.
• Language models: Nearly all modern language models—ChatGPT, Claude, Gemini, DeepSeek, and others—use some form of RL for post-training, especially for developing advanced reasoning capabilities.
• Traffic control: Research on RL-based vehicular control shows significant improvements in traffic flow optimization.
• Generative image models: RL can train diffusion models to better follow text prompts, improving alignment between generated images and user intent.
• Chip design: Google used RL to design the floorplan of production TPU chips, with an agent that sequentially places circuit macros to minimize wirelength, congestion, and density.

3. Fundamental to Intelligence

Learning from experience seems to be a core component of intelligence—both natural and artificial. RL enables the ability to get better with practice. Early robotics work by Levine et al. (ICRA 2015, JMLR 2016) demonstrated this vividly: a robot that initially could not perform basic manipulation tasks learned, through repeated practice, to insert shaped blocks into matching holes—first without vision (using only proprioceptive feedback), then using raw camera images as input. This kind of autonomous skill acquisition through trial and error is something that no amount of supervised training can achieve on its own.

4. Plenty of Exciting Open Research Problems

Despite its successes, deep RL still has major open challenges that represent active areas of research:

• Reward learning: How does an agent learn to represent what is good or bad for a task, without hand-designed reward functions?
• Generalization at scale: How can an agent generalize its behavior to many different scenarios? This motivates offline RL (leveraging large, diverse datasets) and multitask/meta-RL (transferring from other tasks and goals).
• Long-horizon tasks: Can RL learn complex, multi-step behaviors like cooking a meal? This connects to hierarchy and reasoning.
• Autonomous practice: Can robots practice fully autonomously without human resets between episodes? This is the problem of reset-free RL.

A humorous but telling illustration from the early days: behind the scenes of RL robotics experiments, a human researcher ("Yevgen") was often doing more work than the robot—manually resetting objects, monitoring training, and intervening when things went wrong. Making this process scalable and autonomous remains an important goal.

Representing Experience as Data

Before we can formalize the RL problem mathematically, we need a vocabulary for describing the data that arises from sequential decision making. Every RL problem involves an agent interacting with an environment over discrete time steps, and we describe this interaction using a small set of core quantities.

These quantities are highly flexible and can represent a wide variety of problems. The specific meaning of "state," "action," and "reward" varies enormously depending on the domain.

Notice how different these two examples are in their specifics—continuous physical control versus discrete text generation, fixed-length versus variable-length trajectories, dense versus sparse rewards—yet the same abstract framework applies to both.

States, Observations, and the Markov Property

A critical distinction in RL is between states and observations. The state $s_t$ is the complete description of the world at time $t$, containing all information necessary to predict what happens next. The observation $o_t$ is what the agent actually sees, which may be an incomplete or noisy view of the true state.

The key structural property that makes sequential decision-making tractable is the Markov property: the next state depends only on the current state and action, not on the full history of past states and actions.

The graphical model for this structure has states $s_1 \to s_2 \to s_3 \to \cdots$ forming a chain, with observations $o_t$ depending on state $s_t$, and actions $a_t$ depending on the observation. The crucial feature is that the arrows between states form a chain: $s_{t+1}$ depends on $s_t$ and $a_t$, but is conditionally independent of all earlier states given $s_t$.

When the agent can observe the full state ($o_t = s_t$), we have a fully observed setting. When the agent only sees partial information about the state, we have a partially observed setting. In the partially observed case, the agent must either maintain some form of memory (e.g., a recurrent neural network or a history of recent observations) or accept that its decisions will be based on incomplete information.

Policies: Representing Behavior

A policy $\pi$ is the agent's decision-making rule—it determines how the agent selects actions based on what it observes. In deep RL, the policy is typically parameterized by a neural network with parameters $\theta$, written $\pi_\theta$.

The agent's interaction loop with the environment proceeds as follows:

1. Observe state $s_t$ (or observation $o_t$).
2. Take action $a_t$, e.g., by sampling from the policy: $a_t \sim \pi_\theta(\cdot \mid s_t)$.
3. Observe next state $s_{t+1}$, sampled from the unknown world dynamics $p(\cdot \mid s_t, a_t)$.
4. Repeat, producing a trajectory $s_1, a_1, s_2, a_2, \ldots, s_T, a_T$, also called a policy roll-out or an episode.

In the fully observed case, the policy is a conditional distribution $\pi_\theta(a \mid s)$. In the partially observed case, where the agent only sees observations $o_t$ rather than full states, we give the policy memory by conditioning on a window of recent observations:

This can be implemented with recurrent networks, transformers, or simply by stacking recent observations as input. The network architecture maps the state (e.g., an image processed by convolutional layers, followed by fully connected layers) to a distribution over actions.

Why Stochastic Policies?

You might wonder why we define the policy as a probability distribution $\pi_\theta(a \mid s)$ over actions rather than a deterministic mapping $a = f_\theta(s)$. There are two important reasons:

1. Exploration: To learn from its own experience, the agent must try different things. A deterministic policy would always take the same action in a given state, never discovering potentially better alternatives. Stochasticity provides a natural mechanism for exploration.
2. Modeling stochastic behavior: Real-world data often exhibits varying behaviors—different human demonstrators may take different actions in the same situation. A stochastic policy can capture this variability.

This connection to generative modeling is powerful: a policy $\pi_\theta(a \mid s)$ is simply a conditional generative model over actions given states or observations. All the tools from modern generative modeling—normalizing flows, diffusion models, autoregressive models—can be brought to bear on learning policies.

The Reinforcement Learning Objective

With the concepts of states, actions, rewards, and policies in hand, we can now state the goal of reinforcement learning precisely.

Maximizing the Sum of Rewards

The most natural objective is to choose a policy that maximizes the total reward accumulated over a trajectory:

However, there is an immediate problem: this sum is not a deterministic quantity. Even with a fixed policy $\pi_\theta$, different roll-outs will produce different trajectories and different total rewards. What are the sources of this variability?

1. The world is stochastic: The transition dynamics $p(s_{t+1} \mid s_t, a_t)$ may be random. In autonomous driving, for example, other drivers behave unpredictably.
2. The policy may be stochastic: The agent may not make the same decision every time it encounters the same state, especially if using a stochastic policy for exploration.

The Expected Return Objective

To handle this randomness, we optimize the expected sum of rewards:

Here $p_\theta(\tau)$ is the distribution over trajectories induced by the policy $\pi_\theta$ interacting with the environment. Using the Markov property and the chain rule of probability, we can write this trajectory distribution explicitly:

This factorization is a direct consequence of the Markov property. Notice that the trajectory distribution $p_\theta(\tau)$ depends on $\theta$ only through the policy terms $\pi_\theta(a_t \mid s_t)$—the dynamics $p(s_{t+1} \mid s_t, a_t)$ are properties of the environment, not of the agent. This separation is crucial: it means we can optimize the policy without knowing the dynamics explicitly (though knowing or learning them can help).

Value Functions and Q-Functions

The RL objective asks us to find the policy that maximizes expected cumulative reward. A natural question is: how good is a particular policy? Value functions provide the answer by quantifying the expected future reward from a given state (or state-action pair) under a specific policy.

The value function tells us how good it is to be in a state, while the Q-function tells us how good it is to take a particular action in a state. These two quantities are intimately related:

That is, the value of a state under policy $\pi$ is the expected Q-value over the actions that $\pi$ would select. Value functions and Q-functions are central to many RL algorithms. They serve as building blocks for evaluating policies, comparing alternatives, and driving policy improvement.

The Markov Decision Process Framework

Putting all the pieces together, we arrive at the formal mathematical framework that underlies nearly all of reinforcement learning: the Markov Decision Process (MDP).

The MDP and POMDP are the canonical formalisms for sequential decision-making under uncertainty. The Markov property—that $s_{t+1}$ depends only on $(s_t, a_t)$ and not on the earlier history—is what gives these frameworks their tractable recursive structure.

The goal within this framework is to learn a policy $\pi_\theta$ that maximizes the expected sum of rewards:

All the methods we will study in this course are different approaches to solving this optimization problem, each making different trade-offs and assumptions.

Types of Deep RL Algorithms

Given the RL objective $\max_\theta \, \E_{\tau \sim p_\theta(\tau)} \left[ \sum_t r(s_t, a_t) \right]$, there are several fundamentally different approaches to finding a good policy. Each has its own strengths, weaknesses, and domains of applicability.

1. Imitation Learning

Rather than optimizing the reward objective directly, imitation learning approaches the problem by mimicking a policy that is known to achieve high reward. Given demonstrations from an expert, the agent learns to reproduce the expert's behavior. This sidesteps the need for reward engineering and exploration, but requires access to high-quality demonstrations. We will study this approach in depth in the next lecture.

2. Policy Gradients

Policy gradient methods directly differentiate the expected return objective with respect to the policy parameters $\theta$. By estimating the gradient $\nabla_\theta \, \E_{\tau \sim p_\theta(\tau)} [\sum_t r(s_t, a_t)]$ from sampled trajectories, these methods can optimize any differentiable policy architecture. They are conceptually clean and widely applicable, but can suffer from high variance in gradient estimates.

3. Actor-Critic

Actor-critic methods combine policy optimization with value function estimation. The "critic" estimates the value of the current policy (typically $\Vpi$ or $\Qpi$), and the "actor" uses this estimate to update the policy in a direction that improves performance. This reduces the variance of policy gradient estimates by replacing high-variance reward samples with lower-variance value function estimates.

4. Value-Based Methods

Value-based methods do not maintain an explicit policy at all. Instead, they estimate the value of the optimal policy (typically $\Qstar(s, a)$) and derive actions by selecting the action with the highest estimated value. Q-learning and its deep variant DQN are the canonical examples. These methods work well with discrete action spaces but are harder to apply in continuous settings.

5. Model-Based Methods

Model-based methods learn a model of the environment dynamics $p(s_{t+1} \mid s_t, a_t)$ and then use this model for planning or for improving the policy. By "imagining" trajectories in the learned model, the agent can evaluate potential actions without expensive real-world interaction. These methods can be very data-efficient, but their performance depends on the accuracy of the learned model.

Why So Many Algorithms?

Different algorithms thrive under different assumptions and constraints. The choice depends on several practical considerations:

• Data collection cost: How easy or cheap is it to collect data with the current policy? If you have a fast simulator, online methods are attractive. If data collection is expensive (e.g., real robots), offline or model-based methods may be preferred.
• Supervision availability: Are expert demonstrations available? Is a detailed reward function easy to specify?
• Stability and ease of use: Some algorithms are more stable and easier to tune than others.
• Action space structure: Is the action space discrete (e.g., choosing from a menu of options) or continuous (e.g., specifying joint torques)? High-dimensional or low-dimensional?
• Dynamics complexity: Is the environment dynamics easy to model, making model-based approaches viable?

Summary and Looking Ahead

This lecture established the foundational vocabulary and framework for deep reinforcement learning. Let us recap the key definitions.

We represent sequential decision-making problems using states $s_t$ (or observations $o_t$), actions $a_t$, reward functions $r(s, a)$, and trajectories $\tau = (s_1, a_1, \ldots, s_T, a_T)$. The Markov property states that the next state depends only on the current state and action, giving rise to the Markov Decision Process (MDP) framework with its initial state distribution $p(s_1)$ and unknown dynamics $p(s_{t+1} \mid s_t, a_t)$.

A policy $\pi_\theta$ represents the agent's behavior, mapping states to a distribution over actions. The goal of RL is to learn a policy that maximizes the expected sum of rewards:

The value function $\Vpi(s)$ measures the expected future reward from a state under policy $\pi$, while the Q-function $\Qpi(s, a)$ measures the expected future reward from a state-action pair. These quantities are central to many algorithmic approaches.

We surveyed five major families of algorithms—imitation learning, policy gradients, actor-critic, value-based, and model-based methods—each making different trade-offs appropriate for different problem settings.

In the next lecture, we begin our study of specific algorithms with imitation learning: how to learn a policy from expert demonstrations, the challenges that arise from distribution shift, and techniques like DAgger that address them.