Local Computation for Better Decisions

Every algorithm we've studied so far—value iteration, policy gradient, Q-learning—computes a policy over the entire state space. Before the agent takes a single action, it has reasoned about every state it could ever visit.

That works beautifully until the state space becomes astronomical. In Go, played on a $19 \times 19$ board, there are roughly $10^{170}$ legal positions—more than the number of atoms in the observable universe. No computer can store a value function over this space, let alone compute one. And Go isn't an outlier: robotics, logistics, and real-time strategy games all have state spaces that dwarf any global method.

The idea that rescues us: local, simulation-based planning. Instead of solving the entire MDP, we invest computation at decision time to make a better choice for the current state only. Given a simulator $\mathcal{M}_\nu$, we imagine possible futures from $s_t$, estimate action values, and pick the best one. At the next state, we start fresh.

This shift—from global policy computation to local, on-demand planning—is the foundation of Monte Carlo Tree Search.

Simple Monte Carlo Search

Let's start with the simplest version. We have a model $\mathcal{M}_\nu$ and some simulation policy $\pi$ (could be random, heuristic, or learned). To decide what to do at $s_t$: try each action, run $K$ simulated rollouts under $\pi$, average the returns, and pick the action that looked best.

For each action $a \in \mathcal{A}$, simulate $K$ episodes from the current state:

We then estimate the action-value via Monte Carlo averaging:

and select the real action greedily:

Why does this work? By the law of large numbers, the Monte Carlo average converges to $q_\pi(s_t, a)$ as $K \to \infty$. And by the policy improvement theorem, acting greedily on $q_\pi$ is guaranteed to be at least as good as $\pi$ itself. Even a mediocre simulation policy produces a strictly better decision through search.

But there's a critical weakness: each rollout runs all the way to the terminal state. In Go, that's ~300 moves per player. Even a fast simulator struggles to get reliable estimates. Worse, $\pi$ may be poor in the early moves near $s_t$—exactly where the choice matters most.

We need a way to focus computation on the near-term decisions while still reasoning about long-term outcomes.

Forward Search and Expectimax Trees

Can we do better than one step of improvement? In principle, yes. Given a model $\mathcal{M}_\nu$, we can build a full expectimax tree rooted at $s_t$. The idea: expand all possible actions and successor states to some depth $H$, then work backward from the leaves to compute optimal values at the root.

The tree alternates between two kinds of nodes: decision nodes where the agent picks an action (maximize), and chance nodes where the environment samples a next state (average over outcomes). Computing the optimal value at the root gives us the best action without solving the whole MDP.

The catch? The full tree scales as $(|\mathcal{S}| \cdot |\mathcal{A}|)^H$. For Go with ~250 legal moves and ~300 remaining moves, that exceeds $10^{700}$. Utterly intractable.

We need a method that builds its tree selectively—focusing on promising branches instead of exhaustively enumerating everything. That's exactly what MCTS does.

Monte Carlo Tree Search

The core idea is simple: instead of expanding the entire tree, grow it incrementally. Each simulation traverses the existing tree, extends it by one node, evaluates the new leaf, and propagates the result back up. Over many simulations, the tree grows dense where things look promising and stays sparse where they don't.

Each simulation runs three phases:

1. Select—Starting from the root, traverse the existing tree by choosing actions according to a tree policy (such as UCT, discussed below). The tree policy balances exploration of under-visited branches with exploitation of branches that appear promising. Continue until reaching a leaf of the current tree.
2. Expand and Evaluate—Add one or more new child nodes to the leaf, extending the tree by one layer. Evaluate the new node to obtain a value estimate. In classical MCTS, this evaluation is done by running a rollout—a complete simulation from the new node to a terminal state using a fast default policy (often random). The return from this rollout serves as the value estimate.
3. Backup—Propagate the evaluation result back up the tree along the path traversed during selection. At each ancestor node, update the visit count $N(s, a)$ and the mean action-value $Q(s, a)$.

After $K$ simulations, select the real action at the root with the highest visit count (or highest mean value):

The magic is in the tree policy used during selection. A good tree policy must balance exploration (visiting under-explored branches) with exploitation (focusing on promising ones). Sound familiar? It's exactly the multi-armed bandit problem from Lectures 10–11. The solution that turned MCTS into a powerhouse is called UCT.

Upper Confidence Trees (UCT)

Here's the key connection: at each node in the search tree, the agent faces a multi-armed bandit problem. The "arms" are the available actions, and the "rewards" are returns from simulations passing through each action. UCT (Kocsis and Szepesvári, 2006) simply applies UCB1 at every node.

At node $i$, UCT picks the action with the highest score:

where:

• $N(i, a)$ is the number of times action $a$ was selected at node $i$
• $G_k(i, a)$ is the $k$-th return from node $i$ after taking action $a$
• $N(i) = \sum_a N(i, a)$ is the total visit count at node $i$
• $c$ is an exploration constant controlling the exploration–exploitation tradeoff

For each simulated episode $k$, the tree policy selects the action with the highest upper bound at every node along the path from root to leaf:

Notice that the tree policy evolves across simulations. Early on, many actions have low visit counts and large bonuses—the tree explores broadly. Later, the exploitation term dominates, and simulations concentrate on the best lines of play. This adaptive behavior is a natural form of iterative deepening focused on the highest-value regions.

Kocsis and Szepesvári proved that UCT converges to the optimal action as simulations grow. In practice, it produces strong play with far fewer simulations than a full tree would require.

Advantages of MCTS

What makes MCTS so versatile? A few key properties:

• Selective search — allocates simulations to the most promising branches, not wasting time on clearly inferior actions.
• Dynamic evaluation — evaluates states through simulation rather than a pre-computed heuristic, adapting to novel situations on the fly.
• Sampling over enumeration — avoids exponential blowup by growing the tree selectively in reachable, relevant regions.
• Black-box compatible — only needs to simulate trajectories, not access explicit transition probabilities. Works with physics engines, game simulators, procedural dynamics.
• Anytime and parallelizable — can stop at any point and return the best action found so far. Simulations are largely independent, so they parallelize naturally.

When is MCTS a good fit? Large state spaces, long horizons, and access to a simulator. Games, combinatorial optimization, and planning problems where the model is known or learnable. For small state/action spaces with short horizons, exact methods may be preferable.

Case Study: The Game of Go

The domain that propelled MCTS from curiosity to household name: the ancient game of Go. Roughly 2,500 years old, widely regarded as the hardest classic board game, and identified by John McCarthy as a grand AI challenge. For decades, the strongest Go programs played at a weak amateur level—far behind superhuman chess engines.

Why so hard? The game tree is staggeringly large. Each position has ~250 legal moves (vs. ~35 in chess), and games last ~150 total moves. The tree has roughly $250^{150} \approx 10^{360}$ nodes. Minimax with alpha-beta pruning conquered chess in the 1990s, but it can't tame a branching factor of 250.

An interesting conceptual question: is Go really an RL problem? The rules are perfectly known—it's deterministic and fully observable. The challenge isn't learning the environment but planning within it. Go is fundamentally a planning problem—yet RL techniques proved essential for cracking it.

MCTS was the first breakthrough, bringing computer Go to competitive amateur levels in the mid-2000s. Programs like MoGo, Crazy Stone, and Fuego used UCT with hand-crafted rollout policies to achieve strong amateur play on the full $19 \times 19$ board. But the leap to superhuman required combining MCTS with deep neural networks—the subject of our next lecture.

Summary

We built Monte Carlo Tree Search from the ground up, starting from the question: what if we plan locally instead of globally?

Simple Monte Carlo search gives us one step of policy improvement—simulate $K$ rollouts per action, average the returns, act greedily. Expectimax trees give optimal lookahead but scale catastrophically as $(|\mathcal{S}| \cdot |\mathcal{A}|)^H$. MCTS splits the difference: grow the tree selectively through select → expand & evaluate → backup, focusing simulations where they matter.

The breakthrough ingredient is UCT—applying UCB1 at each tree node to balance exploration and exploitation, turning MCTS into a highly selective best-first search. Go demonstrated the power of this approach: a game tree of $\sim 10^{360}$ nodes, previously intractable, yielded to MCTS at the amateur level.

Next up: how AlphaGo and AlphaZero combined MCTS with deep neural networks and self-play to achieve superhuman performance—replacing random rollouts with learned evaluations and training entirely without human data.