Lecture 18

Frontiers of Deep RL

Open problems and future directions in deep reinforcement learning research.

Frontiers Open Problems Future Directions Research
Original PDF slides

The Frontier Landscape

This final lecture surveys the most important open problems and future directions in deep reinforcement learning research. Over the preceding seventeen lectures, we have built a comprehensive toolkit: imitation learning, policy gradients, actor-critic methods, value-based approaches, model-based RL, offline RL, multi-task learning, RL for LLMs, and sim-to-real transfer. The question now is: where does the field go from here?

The open problems are organized along three axes that mirror the structure of any applied RL project:

  1. Problem setup — defining what we want the agent to optimize, especially in domains where reward signals are ambiguous, delayed, or absent
  2. Methods — developing algorithms that can leverage prior knowledge, use world models effectively, and scale to the computational demands of real-world applications
  3. Deployment and evaluation — ensuring that learned policies are safe, reliable, and can be evaluated before and after deployment

We then conclude with practical advice on how to do research in deep RL—choosing problems, managing risk, sharing results, and building confidence as a researcher.

Non-Rewarding, Non-Verifiable Domains

Many of the most impressive RL successes occur in domains where rewards are well-specified and verifiable: games with clear win/loss conditions, mathematical reasoning with checkable proofs, coding with executable test cases. In these settings, the agent can be evaluated precisely and at scale. But some of the most important real-world applications do not have this luxury.

LLMs for Chatbots

The dominant approach for aligning language models with human preferences is preference optimization—training the model using human comparisons of response pairs. But this approach has deep structural problems. When presented with two responses, human raters tend to prefer outputs that agree with them, that sound confident, and that tell them what they want to hear—regardless of whether the response is actually true. This conflation of agreeableness with quality creates a systematic bias in the reward signal.

The challenges compound further. Individual preferences are inherently subjective, raising tensions between personalization (adapting to each user's preferences) and polarization (reinforcing existing beliefs). Multiple objectives—helpfulness, honesty, harmlessness, conciseness—may compete with each other, and current approaches to balancing them rely on crude heuristics.

Robotics

In robotics, reward functions are often binary (success or failure) or heavily hand-shaped with dozens of manually tuned terms. Consider the task of folding a shirt: how would you score each of three different folds? The "right" answer depends on context, aesthetics, and personal preference in ways that are extremely difficult to formalize as a scalar reward. Current robotics rewards lack the nuance needed for truly general-purpose manipulation.

Recommendation Systems

Platforms like YouTube optimize a weighted combination of engagement metrics (clicks, watch time) and satisfaction signals (likes, surveys). The weights on these competing objectives are manually tuned—a process that does not scale and is sensitive to unforeseen interactions between metrics. Small changes in reward weights can have large, unpredictable effects on user behavior at scale.

Scientific Experimentation and Learning

Perhaps the most ambitious frontier is applying RL to domains where even defining the reward function requires deep domain expertise: scientific experimentation, creative design, and open-ended learning. The success of RL for mathematical reasoning suggests that similar techniques could work for scientific reasoning, but the verification of scientific claims is far more complex than checking a proof.

Key Insight
The domains where RL is most needed—open-ended real-world tasks with subjective, delayed, or unverifiable outcomes—are precisely the domains where reward specification is hardest. Closing this gap is arguably the single most important open problem in applied deep RL. Solutions may involve learning reward models from diverse human feedback, using foundation models as reward proxies, or developing entirely new frameworks that go beyond scalar reward maximization.

Leveraging Prior Data and Knowledge

The default approach to incorporating prior knowledge into RL is straightforward: initialize model weights from a pre-trained model, and initialize the replay buffer with offline data. Methods like those of Ball et al. (2023) demonstrate the value of seeding the training process with demonstrations, initializing only the buffer (not the weights) to avoid constraining the learned policy to the pre-training distribution.

But this raises two deeper questions that remain largely unresolved:

Abstract Prior Knowledge

How can we leverage abstract prior knowledge—hints, knowledge from textbooks or news articles, common-sense physical reasoning—rather than just model weights and trajectories? Current RL algorithms have no mechanism for ingesting unstructured knowledge. A human learning to drive can benefit from reading about traffic rules, watching videos, and hearing advice from experienced drivers; an RL agent can only learn from its own interaction data or pre-collected trajectories in the same format.

Going Beyond Pre-Training

Do pre-training weights and data constrain learning too much? If an LLM is initialized from pre-training and then fine-tuned with RL, can it discover solutions that no human has ever written down? Can a robot policy initialized from demonstrations learn to perform tasks faster and more reliably than the human demonstrators? The evidence from AlphaGo suggests yes—RL can transcend the distribution of its training data—but achieving this consistently across domains remains an open challenge.

Example
Transcending the training distribution. AlphaGo's "Move 37" against Lee Sedol was a move that no human Go player had ever considered. It emerged from RL self-play that explored beyond the distribution of human games. Similarly, the locomotion gaits discovered by RMA (Lecture 17) include hip-abduction strategies on stairs that were never programmed or demonstrated. These examples suggest that RL's ability to go beyond prior data is not just theoretically possible but practically achievable—given the right problem structure and sufficient computation.

Using World Models

Video generation models represent an exciting potential source of world knowledge for RL. Modern video generators (like those from Google's Veo, OpenAI's Sora, and others) have learned rich models of how the visual world evolves over time. They can predict plausible future frames given an initial state, implicitly encoding knowledge about physics, object permanence, and scene dynamics. The question is: how can RL leverage this knowledge?

The most natural approach is to use a video generation model as a learned simulator: given a state $s_t$ and a sequence of actions $a_{t:t+h}$, predict the future state sequence $s_{t+1:t+h}$, then evaluate whether the predicted outcome is desirable. But this approach faces large, nuanced challenges.

The Distribution Shift Problem

If the world model is trained on demonstrations and one policy's rollouts, it has learned the dynamics under a specific distribution of actions. When a new policy produces actions $\tilde{a}_{t:t+h}$ that differ from the training distribution, the predicted future states will be unreliable—these actions are out of distribution for the model. Additionally, small physical inaccuracies in the predicted dynamics compound over time, leading to poor performance even for in-distribution action sequences.

Two possible solutions have been proposed:

  1. Train on data from more policies: By collecting data under diverse policies, the world model can learn dynamics that are valid across a broader range of actions. This is expensive but conceptually straightforward.
  2. Use the model differently: Rather than conditioning on actions, train the model to predict future states from the current state alone (i.e., $s_t \to s_{t+1:t+h}$), trained only on demonstrations. Then generate a predicted future video and use a goal-conditioned policy to reach the predicted desirable future state. This avoids the action-conditioning distribution shift entirely, at the cost of requiring a separate goal-reaching policy.
Key Insight
Video generation models encode rich world knowledge, but using them as drop-in replacements for physics simulators does not work well due to distribution shift and compounding errors. The most promising approaches use these models indirectly—as sources of subgoals, reward signals, or data augmentation rather than as direct dynamics models for planning.

How to Scale Up

Large-scale RL for language models is one of the most exciting developments in modern AI. Yet current approaches have significant limitations that constrain where and how RL can be applied at scale.

Current Limitations

Current large-scale RL practice is either short-horizon or very online:

The central open question is: can we do large-scale RL with longer horizons and less online data?

Accurate Value Functions at Scale

Algorithms like PPO use value functions primarily for reducing gradient variance—the value function baseline subtracts a state-dependent term from the reward-to-go, reducing the variance of policy gradient estimates. But for actor-critic algorithms that rely on value functions for policy improvement (not just variance reduction), the value function must be accurate enough to reliably rank different actions. Training accurate value functions at the scale of modern LLMs—where the state space is the space of all possible conversation histories—remains an open challenge.

Batch Online RL

In many applications, it is impractical to interleave model updates with data collection in a tight loop. Collecting dialogue data with real users, gathering robot experience on physical hardware, or running experiments in the real world all have latency and cost constraints that prevent the rapid iteration typical of online RL.

A more practical paradigm is batch online RL: alternate between collecting a large batch of data with the current policy and updating the model, potentially asynchronously. This paradigm brings new considerations—for example, the policy must be expressive enough to generate diverse data within each batch, ensuring sufficient coverage of the state-action space for meaningful policy improvement.

Definition
Batch Online RL. A training paradigm that alternates between two phases: (1) collecting a large batch of trajectories using the current policy, and (2) performing one or more policy updates on the collected batch. Unlike fully online RL, data collection and model updates are decoupled, allowing for asynchronous and distributed implementations. Unlike offline RL, the data distribution is periodically refreshed with on-policy data.

Safety in Safety-Critical Domains

How should we approach AI development and testing in domains where failures can cause serious harm—medicine, autonomous driving, mental health counseling, legal and political discourse? This is not merely a technical question but a fundamental challenge at the intersection of engineering, ethics, and policy.

The Limits of Formal Guarantees

Historically, safety in engineering has been approached through formal verification and probabilistic guarantees. A bridge is designed with safety factors; a flight control system is verified against a formal specification. But these approaches make assumptions about the operating conditions that will not hold in the open world. A self-driving car encounters situations that no specification anticipated. An AI counselor receives queries that no test suite covered.

It is generally impossible to guarantee safety across the countless scenarios encountered in open-world environments. Even human professionals—drivers, surgeons, pilots, politicians—make mistakes. The question is not whether an AI system will ever fail, but how to make failures rare, detectable, and recoverable.

Large-Scale ML as a Safety Tool

Arguably, large-scale machine learning is the most successful approach we have for handling open-world circumstances. ML systems can generalize to novel situations in ways that rule-based systems cannot. But training safe ML systems requires data about unsafe circumstances, and collecting such data can itself cause harm—a fundamental tension that has had serious real-world consequences.

Two promising research directions may help resolve this tension:

  1. Learning about unsafe situations without expansive unsafe data: Can we develop methods that learn to recognize and avoid dangerous states using synthetic data, transfer from related domains, or prior knowledge about the structure of harm? Simulation provides one avenue, but simulating rare catastrophic events realistically is itself a major challenge.
  2. Safe exploration: Can an agent gradually explore new behaviors while maintaining safety constraints? This is the problem of constrained RL—maximizing performance subject to the constraint that certain catastrophic outcomes never (or almost never) occur. Balancing exploration (trying new things) with exploitation (sticking to known-safe behaviors) takes on a new urgency when the cost of a bad exploration step is high.
Key Insight
Safety in RL cannot be achieved through reward engineering alone. A negative reward for unsafe actions presupposes that the agent can recognize unsafe situations, which is precisely the hard problem. The most promising approaches combine multiple lines of defense: conservative initial policies, constrained optimization, human oversight during deployment, and continuous monitoring for distributional shift.

Handling Inaccuracies and Hallucinations

When AI systems interact with humans—as chatbots, code assistants, or medical decision support tools—mistakes are inevitable. The question is how to handle them well. Recent evidence suggests that current AI systems are not well optimized for human-AI collaboration.

Optimizing Human-AI Systems

A striking finding reported by Rajpurkar and Topol (NYT, 2025; Goh et al., JAMA, 2024) illustrates the problem: when an AI system worked independently to diagnose patients, it achieved 92% accuracy. When physicians used the AI as an assistant, their combined accuracy was only 76%—barely better than the 74% they achieved without AI. The AI actually degraded the physicians' performance, likely because the interface did not effectively communicate the AI's confidence and reasoning.

This suggests that optimizing the AI model alone is insufficient. We need to optimize the entire human-AI system, including how the model communicates its uncertainty to the human decision-maker.

Calibration and the RLHF Problem

One crucial property for effective human-AI collaboration is calibration: when a model says it is 80% confident, it should be correct about 80% of the time. Well-calibrated models enable humans to make informed decisions about when to trust the AI and when to rely on their own judgment.

An important empirical finding is that RLHF post-training hurts model calibration. Pre-trained GPT-4 exhibits reasonably good calibration—its confidence scores correlate well with its actual accuracy across different confidence levels. After PPO-based RLHF training, this calibration degrades significantly. The model becomes overconfident, producing high-confidence predictions even when it is wrong. This is a fundamental tension: RLHF optimizes for human preference, and humans prefer confident-sounding answers, creating an incentive for the model to be overconfident.

Example
Calibration before and after RLHF. Consider a pre-trained language model that, when it assigns 70% confidence to an answer, is correct about 70% of the time. After RLHF training, the same model might assign 90% confidence to answers it is only correct about 70% of the time. A human using this model would over-trust its predictions, potentially making worse decisions than they would with a less confident but better-calibrated system. This is precisely the pattern observed with GPT-4 before and after PPO post-training (Tian et al., 2023; OpenAI, 2023).

Possible solutions include training models to express uncertainty through verbalized confidence ("I'm about 60% sure that...") and listing multiple plausible answers rather than committing to a single response. These approaches preserve the benefits of RLHF-trained fluency while giving humans the uncertainty information they need.

The Path to 99.99% Reliability

When humans are not in the loop—fully autonomous systems—the bar for reliability becomes much higher. Getting from 90% to 99.99% reliability is a qualitatively different challenge from getting from 0% to 90%. Reinforcement learning is likely part of the solution, with promising results in some scenarios (Luo et al., 2024), but achieving near-perfect reliability in open-world environments remains one of the hardest unsolved problems in AI.

Evaluation of Generalist Systems

In supervised learning, evaluation is conceptually straightforward: measure accuracy on a held-out validation set drawn from the same distribution as the training data. In reinforcement learning, this approach fundamentally breaks down.

Why Offline Evaluation is Hard

The core difficulty is that RL data is collected under a policy that differs from the learned policy. A held-out dataset collected by an old policy tells us about the performance of that old policy, not the new one. The new policy will visit different states, take different actions, and encounter different outcomes. Evaluating a policy on states it will actually visit requires running it in the real environment—but for many applications, this is exactly what we are trying to avoid before we are confident the policy is safe and effective.

This problem is exacerbated for generalist systems that must perform well across many conditions. A self-driving car must handle urban streets, highways, construction zones, rain, snow, night driving, and countless other scenarios. Evaluating performance across this combinatorial space through real-world testing alone is infeasible.

Open Questions in Evaluation

Two critical open questions define the frontier of RL evaluation research:

  1. Can we develop offline metrics that at least rule out bad models? Even if we cannot precisely estimate a policy's real-world performance from offline data, can we develop metrics that reliably identify policies that will fail? Such conservative screening metrics would be enormously valuable for filtering candidate policies before expensive real-world evaluation.
  2. How should we select representative real-world scenarios for online evaluation? Given that we cannot test every possible scenario, which scenarios should we prioritize? This is a problem of experimental design for policy evaluation—a field that is currently far less developed than the policy optimization methods it should support.
Key Insight
Evaluation may be the most underappreciated bottleneck in deploying RL systems. We have powerful algorithms for training policies, but our ability to verify that those policies will work before deployment lags far behind. The evaluation gap is particularly severe for generalist systems, where the space of possible operating conditions is vast and the cost of failure is high. Developing reliable evaluation methodology is as important as developing better training algorithms.

How to Do (Deep RL) Research

The second half of this lecture shifts from open technical problems to the practice of research itself. These insights are drawn from the instructor's own experience and are offered with the caveat that a diversity of research approaches is healthy—there is no single correct way to do research.

Some Realities

Before discussing strategy, it is important to set realistic expectations about the research process:

What to Work On

Choosing a research problem is a three-step process:

  1. Find the intersection of an important problem and a feasible plan. Having only an important problem ("solve climate change") without a plan for how to make progress is not enough. Having only a clever technique ("an algorithm that improves robot success rate by 1%") without an important problem to apply it to is also insufficient. You need both.
  2. Check your excitement. Research is a tremendous amount of work. You will be far more successful—and far more persistent in the face of setbacks—if you are genuinely excited about the problem.
  3. Stress-test the idea honestly. If you are brutally honest about why the idea could fail to solve the problem, does it still have a high chance of working? If not, it probably will not work. The most common failure mode is optimism bias—convincing yourself that known difficulties will somehow resolve themselves.

Idea-Driven versus Problem-Driven Research

There are two fundamentally different orientations to research:

Problem-driven research has the advantage of guaranteeing that you are working on something important. And the advice not to box yourself into one area is important: crossing topic boundaries brings both new problems and new ideas. The instructor's own experience illustrates this—as a robotics researcher, they pivoted to developing a better video generation model when they discovered that existing models were inadequate for their robotics goals. The resulting paper became foundational to their later work and has over 1,300 citations.

Handling Risk

Given that most research ideas do not succeed, managing risk is essential:

  1. Front-load the risk. Before building large-scale infrastructure, formulate and run small, targeted experiments that test the core unknowns. This is uncomfortable—it means confronting the possibility of failure early—but it is far better than discovering after months of engineering that the core idea does not work.
  2. Design targeted experiments that test unknowns in the fastest possible way. Every experiment should be designed to answer a specific question.
  3. Try many ideas—including different problems—to "create luck." The more shots on goal, the more likely one will succeed.
  4. Do not mentally commit to a project before seeing signs of life on the core unknown. Premature commitment leads to the sunk cost fallacy.

Getting Things to Work

When a research project is stuck, several strategies can help:

Deciding When to Pivot

The decision to abandon a project is often considered later than it should be, due to the sunk cost fallacy—the tendency to continue investing in a project because of the time already spent rather than because of its future prospects.

A useful reframing is to replace the binary decision ("continue or quit?") with a comparison: "continue working on this project" versus "work on project B" versus "work on project C." When framed as a choice among concrete alternatives rather than as giving up, the decision becomes less anxiety-inducing and more rational. This requires actively spending time thinking about alternative research projects, even when your current project is going well.

Sharing Research and Closing Thoughts

Why Sharing Matters

The output of research is ideas, knowledge, and learnings—almost never a product or service. If no one knows about the learnings, then there was effectively no output. Even large companies invest heavily in marketing and communication; researchers should invest similarly in making their work accessible and visible.

Common objections to self-promotion in research are worth addressing directly:

How to Share Effectively

Effective communication of research requires deliberate practice:

Mentorship and Confidence

Two final pieces of advice that apply broadly to any research career:

Lean on mentorship. If you have access to experienced mentors, do not be afraid to use their guidance. Students often do best when learning gradually, and a good mentor can help calibrate ambition, identify blind spots, and provide perspective during setbacks.

Build confidence deliberately. There are many reasons to lack confidence in research: no one knows the best way to do research right now, no one knows which work will be most impactful in the future, many ideas do not work, and many papers get rejected. Researchers who have been thinking about a domain for years will be "smarter" in that specific area, but that is a reflection of their experience, not their inherent ability—there is no reason to be intimidated. Self-doubt and overthinking can be significant obstacles to productivity and creativity.

Key Insight
The combination of an important problem, a feasible approach, genuine excitement, and honest self-assessment provides a strong foundation for impactful research. The technical skills developed throughout this course—from MDPs and policy gradients to offline RL and sim-to-real transfer—are the tools. The open problems surveyed in this lecture are the opportunities. The research methodology discussed here is the map for navigating between them.

Course Summary

This final lecture surveyed the frontier of deep reinforcement learning across seven open problem areas, then provided practical guidance on research methodology. The key open problems are:

These challenges span the full pipeline from problem formulation to deployment, and solving them will require not just algorithmic innovation but also new evaluation frameworks, safety paradigms, and ways of integrating human judgment with automated optimization. The students of CS 224R are now well-equipped to begin tackling these problems.