What Is a World Model?
Imagine you could practice a task entirely in your head—simulating what would happen if you took each possible action, without ever touching the real world. That's the promise of a world model: a learned simulator of the environment that an agent can query for planning, imagination, and decision-making.
The idea traces back to Jürgen Schmidhuber (1990), who proposed training a "model network" to predict future observations and rewards alongside a "control network" that uses those predictions to plan. Ha & Schmidhuber (2018) popularized the term in their aptly titled "World Models" paper. Control theorists have called the same object a "dynamics model" since the 1960s. Shane Gu's distillation cuts to the chase: "World model is the 'model' in model-based RL."
The field evolved rapidly in the deep learning era:
- 2015: Oh et al. — action-conditional video prediction in Atari games, showing that neural networks can learn to simulate game physics from raw pixels.
- 2016: Finn et al. — unsupervised learning for physical interaction through video prediction, demonstrating that robots can learn object dynamics from self-supervised play.
- 2017: Weber et al. (DeepMind) — Imagination-Augmented Agents (I2A), which augment model-free policies with imagined rollouts from a learned world model.
- 2021: Hafner et al. — DreamerV2, which masters Atari from pixels using a discrete latent world model trained entirely in imagination, achieving human-level performance with a fraction of the environment interactions.
A world model is a learned approximation of environment dynamics — given the current state $s_t$ and action $a_t$, it predicts the next state $s_{t+1}$. In model-based RL, the agent uses the world model to simulate future trajectories ("imagine") without interacting with the real environment, enabling more sample-efficient learning and planning.
What Is Prediction? — Solomonoff Induction
Before we get into the mechanics, Shane Gu steps back to ask a deeper question: what does it even mean to predict well?
The answer comes from Solomonoff induction—the theoretical gold standard for inductive inference. The core idea is elegant: given observed data $D_{t-1}, D_t, D_{t+1}, \ldots$, the best world model is the one that infers the shortest program $P$ that reproduces the data. Think of it as the ultimate "induction machine"—it takes data and infers rules, causes, programs. The reverse direction (program $\to$ data) is generation.
Formally, we apply Bayes' theorem in the space of all computable programs:
$$p(P \mid D) \propto p(D \mid P) \, p(P)$$The prior $p(P)$ implements Occam's razor: shorter programs are exponentially more probable:
$$p(P) \propto 2^{-|P|}$$where $|P|$ is the length of program $P$ in bits. This is deeply connected to Kolmogorov complexity—the length of the shortest program that produces a given string—and the Minimum Description Length principle, which selects models that achieve the best compression of the data.
Why does this matter for RL? Solomonoff induction inspired a formal definition of intelligence itself. Legg & Hutter (2007) define Universal Intelligence as goal-achievement across a Solomonoff-weighted distribution of environments:
$$\Upsilon(\pi) = \sum_{\mu \in E} 2^{-K(\mu)} V_\mu^\pi$$where $K(\mu)$ is the Kolmogorov complexity of environment $\mu$ and $V_\mu^\pi$ is the expected value of policy $\pi$ in that environment. Simpler environments receive more weight—intelligence is measured by performance across all computable environments, weighted by complexity.
Prediction = Understanding
Ilya Sutskever crystallized the implication: "The best prediction is inferring the shortest program that reproduces the data." In other words, prediction = understanding = inferring the causal program of a phenomenon. The evidence keeps accumulating:
- IQ tests and "aha!" moments are essentially compression and induction tasks—find the pattern, infer the rule, predict the next element.
- Grokking in deep learning—sudden generalization after prolonged memorization—appears to be a phase transition from memorizing data to understanding the underlying program.
- Text prediction yields emotional understanding: the OpenAI Sentiment Neuron showed that a model trained solely to predict the next character in Amazon reviews spontaneously learned a representation of sentiment.
- Video and audio prediction yield physical and emotional understanding: Google's Veo3 demonstrates that models trained to predict video frames develop an implicit understanding of physics, object permanence, and even human emotion.
Given observed data $D$, Solomonoff induction computes the posterior over programs via $p(P \mid D) \propto p(D \mid P) \, p(P)$, with an Occam's razor prior $p(P) \propto 2^{-|P|}$ favoring shorter programs. It is the theoretically optimal (but incomputable) framework for inductive inference — the gold standard against which all learning algorithms can be measured.
Prediction as the path to intelligence. If mastering prediction requires inferring compact causal programs, then a sufficiently powerful predictive model is, in a deep sense, an understanding machine. This is why next-token prediction in LLMs yields reasoning, why video prediction yields physics understanding, and why prediction and intelligence may be two sides of the same coin.
Causality and Out-of-Distribution Generalization
If prediction equals understanding, when does prediction fail? When the learned model relies on spurious correlations rather than causal structure. A classifier might learn that cows appear on green backgrounds and camels on sandy ones—but that pattern shatters the moment you show it a cow on a beach.
This is the problem of out-of-distribution (OOD) generalization, formalized by Invariant Risk Minimization (Arjovsky et al., 2019). True generalization requires learning features that are invariant across different environments—features that reflect genuine causal mechanisms rather than accidental covariation.
The practical takeaway is striking: "Diversity is all you need." Training on data from many different environments pushes the model toward causal features, because spurious correlations are environment-specific while causal relationships are invariant. This is exactly why large-scale pretraining works—GPT-3 trained on internet-scale text from millions of sources is forced toward the true causal structure of language rather than shortcuts specific to any one domain.
Why diversity enables generalization. Invariant Risk Minimization shows that spurious correlations are environment-specific — they break when the data distribution shifts. By training on maximally diverse data (many environments, many interventions), a model is forced to learn only the features that are invariant — i.e., the causal ones. This is why massive internet-scale pretraining tends to produce models that generalize: diversity of training data approximates diversity of interventions on the true causal graph.
Empowerment and Three Levels of Prediction
How far can an agent go in its quest to predict the world? Shane Gu introduces a striking taxonomy—three levels of increasing ambition:
Level 1: Passively Fit Your World Model on the World
This is standard pretraining—supervised learning, self-supervised learning, generative modeling. The data distribution is stationary; the agent passively observes and learns to predict. Yann LeCun's "cake" analogy captures the scale: self-supervised learning (millions of bits per sample) is the cake, supervised learning (10–10,000 bits) is the icing, and RL (a few bits) is the cherry on top.
Level 2: Actively Fit Your World Model on the World
Now the agent's actions influence what data it collects—the distribution becomes non-stationary. This encompasses post-training, DAgger, GAIL, active learning, and curiosity-driven exploration. Two examples make this concrete:
- ICM (Pathak et al., 2017): Intrinsic Curiosity Module, which uses prediction error of a forward dynamics model as intrinsic reward. The agent seeks out states where its world model is most wrong—exploring precisely where it has the most to learn.
- VIME (Houthooft et al., 2016): Variational Information Maximizing Exploration, which uses information gain as intrinsic motivation. The agent prefers actions that maximally update its beliefs about the dynamics.
We can make this precise with information gain. The intrinsic reward augments the extrinsic reward with the KL divergence between the posterior before and after observing the transition:
$$I(S_{t+1}; \Theta \mid \xi_t, a_t) = \mathbb{E}_{s_{t+1} \sim \mathcal{P}(\cdot \mid \xi_t, a_t)} \left[ D_{\text{KL}}\!\left[ p(\theta \mid \xi_t, a_t, s_{t+1}) \,\|\, p(\theta \mid \xi_t) \right] \right]$$ $$r'(s_t, a_t, s_{t+1}) = r(s_t, a_t) + \eta \, D_{\text{KL}}\!\left[ p(\theta \mid \xi_t, a_t, s_{t+1}) \,\|\, p(\theta \mid \xi_t) \right]$$Level 3: Actively Fit the World to Your World Model
This is the most ambitious level—the agent takes actions in the real world to make the world itself more predictable. The examples here come not from RL but from human institutions: politicians shaping public opinion, financial firms manipulating markets, social media influencers curating audiences. As Shane Gu notes: "Difficult objective. Nobody has cracked this yet at scale."
Level 3 connects to empowerment (Polani, 2005; Mohamed & Rezende, 2015)—the channel capacity between an agent's actions $z$ and the resulting future states $s$:
$$\mathcal{I}(s; z) \geq \mathbb{E}_{z \sim p(z), s \sim p(s|z)} \left[ \log q_\phi(s \mid z) - \log p(s) \right]$$An empowered agent can reliably cause a wide range of distinct future outcomes. Maximizing empowerment drives unsupervised skill discovery—the agent learns diverse, controllable behaviors without any extrinsic reward signal.
Empowerment is the channel capacity (maximum mutual information) between an agent's actions and the resulting future states: $\mathcal{I}(s; z) = \max_{p(z)} I(S; Z)$. An empowered agent can reliably cause a wide range of distinct future outcomes. Maximizing empowerment drives unsupervised skill discovery — the agent learns diverse, controllable behaviors without any extrinsic reward.
The three levels as a hierarchy of agency. Level 1 (passive prediction) is what LLMs do during pretraining. Level 2 (active data collection) is what RL agents do during exploration. Level 3 (reshaping the world) is what powerful real-world actors do — and what future AI agents may aspire to. Each level requires strictly more agency and has strictly more impact on the data distribution.
Forward and Inverse World Models
Now let's get concrete. A world model can take two fundamental forms. The forward model (dynamics model) maps a current state and action to the next state:
$$s_{t+1} = F(s_t, a_t)$$Inverse models come in two flavors, each serving a distinct purpose:
The inverse dynamics model is policy-like—given the current and next state, it infers the action that connects them:
$$a_t = \Pi(s_t, s_{t+1})$$The inverse constraint model is Q-function-like—it defines a constraint that must be satisfied for a transition to be dynamically feasible:
$$0 = Q(s_t, a_t, s_{t+1})$$Notice the deliberate symbol choices: $\Pi$ because the inverse dynamics model is structurally a policy (maps states to actions), and $Q$ because the constraint function is structurally a Q-function. The consistency conditions between forward and inverse models are:
$$a = \Pi(s, F(s, a)) \quad \forall \; s, a$$ $$0 = Q(s, a, F(s, a)) \quad \forall \; s, a$$A forward model $F$ predicts the next state given current state and action: $s_{t+1} = F(s_t, a_t)$. An inverse dynamics model $\Pi$ recovers the action given two consecutive states: $a_t = \Pi(s_t, s_{t+1})$. An inverse constraint model $Q$ verifies whether a transition is dynamically feasible: $Q(s_t, a_t, s_{t+1}) = 0$ iff the transition is valid. Forward models are used for shooting-based planning; inverse models enable collocation-based planning.
Shooting vs. Direct Collocation for Planning
We have a world model. Now what? How do we actually use it to plan? Two classical approaches from optimal control offer fundamentally different strategies.
Shooting is the more intuitive approach: optimize over action sequences and use the forward model to roll out trajectories:
$$\arg\max_{a_{t:t+T}} \sum_{i=t}^{t+T} r(s_i) \quad \text{where} \quad s_{i+1} = F(s_i, a_i)$$The forward model is "in the loop"—each candidate action sequence is simulated forward through the dynamics. This is the standard approach in model-based RL: Model Predictive Control (MPC), the Cross-Entropy Method (CEM), and related shooting methods all optimize actions while the model handles the state evolution.
Direct collocation optimizes over both state and action sequences simultaneously, enforcing dynamics as constraints rather than simulating forward. Using the inverse dynamics ($\Pi$) model:
$$\arg\max_{s_{t:t+T}} \sum_{i=t}^{t+T} r(s_i) \quad \text{s.t.} \quad -|A| \leq \Pi(s_i, s_{i+1}) \leq |A|$$Using the inverse constraint ($Q$) model:
$$\arg\max_{s_{t:t+T}, a_{t:t+T}} \sum_{i=t}^{t+T} r(s_i) \quad \text{s.t.} \quad Q(s_i, a_i, s_{i+1}) = 0$$Contact Invariant Optimization
Mordatch et al. (2012, SIGGRAPH) demonstrated the power of direct collocation for contact-rich planning with a principle that sounds almost paradoxical: "First solve the task, then fix the physics." By relaxing the dynamics constraints during optimization, the planner can discover solutions to contact-rich locomotion and manipulation tasks that shooting methods struggle with.
The physics loss penalizes violations of the equations of motion, while the contact invariant loss ensures that contact forces are only applied at contact points:
$$\tau(\mathbf{q}, \dot{\mathbf{q}}, \ddot{\mathbf{q}}) = J(\mathbf{q})^T \mathbf{f} + B\mathbf{u}$$ $$L_{\text{Physics}}(\mathbf{s}) = \sum_t \left\| J_t(\mathbf{s})^T \mathbf{f}_t(\mathbf{s}) + B\mathbf{u}_t(\mathbf{s}) - \tau_t(\mathbf{s}) \right\|^2$$ $$L_{\text{CI}}(\mathbf{s}) = \sum_t c_{i,\phi(t)}(\mathbf{s}) \left( \|\mathbf{e}_{i,t}(\mathbf{s})\|^2 + \|\dot{\mathbf{e}}_{i,t}(\mathbf{s})\|^2 \right)$$The optimizer begins with loose physics constraints, finds a task solution in this "easy" unconstrained space, then gradually tightens the constraints to recover physical plausibility. Collocation solves contact-rich tasks with minimal reward shaping because dynamics relaxation acts as implicit curriculum.
An instructive analogy: shooting is like autoregressive generation (each step depends on the previous), while collocation is like bidirectional diffusion (all timesteps are optimized jointly). This parallel is not merely cosmetic—it reflects a deep structural similarity in how information flows through the optimization.
Shooting optimizes actions and simulates forward through the dynamics: $\arg\max_{a_{t:T}} \sum r(s_i)$ where $s_{i+1} = F(s_i, a_i)$. Direct collocation optimizes states (and optionally actions) jointly, enforcing dynamics as constraints. Shooting is simpler but suffers from long-horizon credit assignment; collocation handles contact-rich tasks by temporarily relaxing physics constraints.
"First solve task, then fix physics." Contact Invariant Optimization (Mordatch et al., 2012) demonstrates a powerful principle: by relaxing dynamics constraints during planning, the optimizer can first find a task solution in an "easy" unconstrained space, then gradually tighten the constraints to recover physical plausibility. This is analogous to how bidirectional diffusion models can generate globally coherent sequences, while autoregressive models must commit sequentially.
Goal-Conditioned Value Functions and Decision Transformers
Remember those deliberate symbol choices—$\Pi$ for policy, $Q$ for constraint? That notation hinted at a deeper connection: inverse models are goal-conditioned policies and value functions.
Temporal Difference Models
Temporal Difference Models (Pong et al., 2018, ICLR) reinterpret a goal-conditioned Q-function as an implicit world model. The key idea: a Q-function $Q(s_t, a_t, s_g, \tau)$ that predicts whether taking action $a_t$ in state $s_t$ will reach goal $s_g$ in $\tau$ steps is, implicitly, a temporally-extended dynamics model. The reward is sparse—nonzero only when the countdown reaches zero:
$$r_d(s_t, a_t, s_{t+1}, s_g, \tau) = -D(s_{t+1}, s_g) \mathbb{1}[\tau = 0]$$The Q-function satisfies a Bellman-like recursion that elegantly bridges one-step dynamics and long-horizon reachability:
$$Q(s_t, a_t, s_g, \tau) = \mathbb{E}_{p(s_{t+1}|s_t, a_t)}\!\left[ -D(s_{t+1}, s_g) \mathbb{1}[\tau = 0] + \max_a Q(s_{t+1}, a, s_g, \tau - 1) \mathbb{1}[\tau \neq 0] \right]$$Training uses hindsight relabeling—any trajectory can be relabeled as a "successful" attempt to reach the state it actually reached. At $\tau = 1$, the Q-function is a one-step dynamics model; at $\tau = T$, it is a long-horizon reachability predictor. Value functions are world models operating at different time scales.
Direct collocation with TDMs uses the goal-conditioned Q-function as the dynamics constraint in a hierarchical planning framework:
$$a_t = \underset{a_{t:K:t+T}, s_{t+K:K:t+T}}{\arg\max} \sum_{i=t, t+K, \ldots, t+T} r_c(s_i, a_i)$$ $$\text{such that } Q(s_i, a_i, s_{i+K}, K-1) = 0 \;\; \forall \; i \in \{t, t+K, \ldots, t+T-K\}$$Generalized Decision Transformers
Generalized Decision Transformers (Furuta, Matsuo, Gu, ICLR 2022 Spotlight) provide a unifying framework that views many offline RL methods through the lens of hindsight information matching:
- Training: behavioral cloning conditioned on "any future statistics" $\phi_T$—return-to-go, final state, trajectory statistics, or other summaries of what happened after the action.
- Test-time: provide an unseen future statistic (e.g., a high return-to-go) and ask the model to "generalize"—producing behavior that achieves it.
The architecture uses a causal transformer (forward in time) and an anti-causal aggregator (backward in time to compute hindsight statistics). This unifies Decision Transformer (Chen et al., 2021), various goal-conditioned methods, and offline RL approaches under a single framework.
The connection to collocation is deep: a world model is a causal predictor (forward in time), while a Decision Transformer is an anti-causal predictor (conditioned on future outcomes). Hindsight experience replay is the "Jedi mind trick" to flip the causality—turning any trajectory into training signal for the anti-causal direction.
TDMs reinterpret goal-conditioned Q-functions as implicit world models. The Q-function $Q(s_t, a_t, s_g, \tau)$ predicts whether action $a_t$ in state $s_t$ reaches goal $s_g$ in exactly $\tau$ steps. At short horizons, this is a dynamics model; at long horizons, a reachability predictor. Training uses hindsight relabeling: any trajectory provides a positive example for the goal it actually reached.
There's a deep duality here: forward world models are causal predictors (past → future), while Decision Transformers are anti-causal (desired future → actions). Hindsight experience replay bridges the two by retroactively conditioning on observed outcomes, turning every trajectory into anti-causal training data.
Physical and Symbolic World Models — Video Models as Reasoners
So far we've treated world models as mathematical objects—dynamics predictors for planning. But what if a world model could see?
2022 — "AGI Year 0"
Shane Gu marks 2022 as a pivotal year where two breakthroughs converged:
- "LLMs can reason" — Zero-shot chain-of-thought (Kojima, Gu et al., 2022): adding "Let's think step by step" to LLM prompts enables multi-step reasoning without any task-specific fine-tuning. This suggests symbolic reasoning is achievable through language models trained purely on prediction.
- "ImagenVideo / DreamFusion" — high-quality video generation and text-to-3D synthesis suggest physical reasoning is achievable through video models trained on internet-scale visual data.
Together, these results sketch a division of labor: LLMs handle symbolic reasoning via Chain-of-Thought (chains of logical deductions in language), while video models handle spatiotemporal reasoning via Chain-of-Frames (step-by-step visual simulations). The world consists of symbols, space, and time—and we may need both types of models to capture its full complexity.
2025 — Video Models as the Missing Foundation Model
Wiedemer, Li, Vicol, Gu et al. (Google DeepMind, 2025) push this vision further: "Video models are zero-shot learners and reasoners." Just as LLMs went from task-specific NLP to general-purpose language understanding by scaling up, video models may follow the same path for vision and physical understanding. They can already solve maze navigation, pattern completion, and physical reasoning tasks zero-shot—without any task-specific training.
What about combining both modalities? Mind's Eye (Liu et al., 2022) augments chain-of-thought reasoning with a physics simulator (MuJoCo) as a tool. LLMs alone make physical reasoning errors—predicting that heavier objects fall faster, for instance—but augmenting them with a physics engine for "mental simulation" yields correct answers.
This raises a provocative question. Specialized approaches like graph neural networks for fluid simulation (Sanchez-Gonzalez et al., 2020) work well in their domains. But Shane Gu conjectures: "We likely won't need graph nets, NeRF, etc. Just video models." Whether highly structured domains like protein folding will also succumb to general-purpose video prediction remains open.
Shane Gu makes a bold conjecture: sufficiently powerful video models, trained on internet-scale visual data, may learn to simulate physics, chemistry, and biology without domain-specific inductive biases—just as LLMs learned grammar and reasoning from raw text. Whether structured domains like protein folding will also succumb to this trend remains open.
The Future of World Modeling
Where is all of this heading? Shane Gu closes with two speculative frontiers.
Modeling humanity (2026+). What if we could simulate eight billion people? Joon Sung Park's work on generative agents showed that LLM-powered characters exhibit emergent social behavior in simulated villages. The ambition goes further: social simulation, political dynamics, economic systems—world models for the human world.
Modeling financial markets (2026+). AIA Forecaster (Alur, Stadie et al., Bridgewater AIA Labs, 2025) tackles one of the most complex, non-stationary environments imaginable. Multiple agents search for information, make forecasts, calibrate confidence, reconcile disagreements, and produce calibrated predictions. It's agentic world modeling in practice—multiple models collaborating to predict an environment that is actively trying to be unpredictable.
Shane Gu's parting message ties everything together: "Remember Level 3." The most ambitious form of world modeling isn't just predicting the world—it's shaping the world to be predictable.
The ultimate world model. Shane Gu's three levels of prediction form a progression of ambition: passively predict the world (Level 1), actively explore to improve predictions (Level 2), and reshape the world to match your model (Level 3). Most AI research operates at Level 1. The frontier of world modeling — and perhaps of artificial general intelligence — lies at Level 3: agents that don't just model the world but actively shape it.
Summary
We've taken a sweeping tour of world modeling—from philosophical foundations to speculative frontiers.
The deep lesson from Solomonoff induction: prediction = understanding. Optimal prediction requires inferring compact causal programs, which is why next-token prediction yields reasoning and video prediction yields physics understanding. But prediction only generalizes when it captures causal structure rather than spurious correlations—diversity of training data is the key.
Shane Gu's three levels of prediction form a hierarchy of agency: passively fit a model (pretraining), actively collect data to improve it (exploration), and reshape the world itself (empowerment). Most AI research lives at Level 1. The frontier is Level 3.
On the technical side, forward models enable shooting-based planning while inverse models enable collocation—temporarily relaxing physics to find solutions, then tightening constraints. Value functions turn out to be world models at coarser temporal resolution, and Decision Transformers are anti-causal world models trained through hindsight. Looking ahead, video models may become the foundation model for physical reasoning, just as LLMs became the foundation for symbolic reasoning.