Why Go?
Go stood for decades as AI's grand challenge. IBM's Deep Blue conquered chess in 1997 through brute-force search with handcrafted evaluation—but that same approach failed catastrophically in Go. Why?
Scale is part of the answer: chess has ~35 legal moves per position; Go has ~250. A typical chess game lasts ~80 moves; Go ~150. The game tree ($\sim 250^{150}$ nodes) dwarfs anything brute-force can handle. But the difficulty runs deeper. In chess, positions can be evaluated by counting material and assessing structure. In Go, positional judgment is holistic—the value of a stone depends on global patterns of influence and territory that resisted decades of handcrafted heuristics.
From an RL perspective, Go is simply a very large, very hard MDP with delayed, binary reward. The agent must learn a policy mapping board positions to moves that maximizes the probability of winning.
MCTS Advantages (Recap)
As we saw last lecture, MCTS is perfectly suited here: selective search, dynamic evaluation through simulation, sampling instead of exhaustive enumeration, black-box compatible, anytime, and parallelizable. These properties made MCTS the dominant paradigm for computer Go even before deep learning.
What AlphaGo added: a way to make MCTS dramatically more efficient by replacing random rollouts and uniform priors with learned neural network estimates.
AlphaGo Zero's MCTS with Neural Networks
The core technical contribution of AlphaGo Zero (Silver et al., Nature, 2017) is the tight integration of a deep neural network $f_\theta$ with Monte Carlo Tree Search. The network takes a board state $s$ as input and outputs two quantities simultaneously:
$$f_\theta(s) = (\mathbf{p}, v)$$where $\mathbf{p}$ is a probability vector over legal moves (the prior policy) and $v \in [-1, 1]$ is a scalar estimate of the value—the probability of winning from state $s$. This dual-headed architecture is central: a single forward pass provides both the intuition for which moves to consider and the judgment for how good a position is.
The MCTS procedure repeats four phases, typically 1,600 times per move decision:
Phase 1: Select
Starting from the root (the current board position), traverse the tree by selecting at each internal node the action that maximizes a variant of the PUCT (Predictor + Upper Confidence Bound for Trees) formula:
At each internal node during tree traversal, select the action maximizing:
$$a_t = \arg\max_a \left[ Q(s, a) + c_{\text{puct}} \cdot P(s, a) \cdot \frac{\sqrt{\sum_b N(s, b)}}{1 + N(s, a)} \right]$$where:
- $Q(s, a)$ is the mean value of all simulations that passed through action $a$ from state $s$
- $P(s, a)$ is the prior probability from the neural network's policy head
- $N(s, a)$ is the visit count for action $a$ from state $s$
- $c_{\text{puct}}$ is an exploration constant
The first term exploits—prefer actions with high estimated value. The second term explores—prefer actions with high prior probability that have been visited relatively few times. As $N(s,a)$ grows, the exploration bonus shrinks and the algorithm focuses on the empirically best action.
The tree alternates between the agent's moves (maximizing) and the opponent's moves (also maximizing from their perspective), mimicking a minimax structure. At each level, the PUCT formula guides search toward the most promising continuations.
Phase 2: Expand and Evaluate
When a leaf node $s_L$ is reached—a position not yet in the tree—expand it by querying the neural network:
$$(\mathbf{p}, v) = f_\theta(s_L)$$Store the prior probabilities $P(s_L, a) = p_a$ for all legal actions $a$. The value estimate $v$ replaces the need for a random rollout to a terminal state—this is a key difference from vanilla MCTS. Instead of playing out random moves to estimate who wins, a single neural network forward pass provides a learned evaluation that captures strategic understanding far beyond what random play could reveal.
This is the key architectural insight: instead of playing out random moves to guess who wins, a single network forward pass provides a learned evaluation capturing strategic understanding far beyond what random play could reveal.
Phase 3: Backup
Propagate the value $v$ back up the tree, updating statistics for every edge traversed during the selection phase:
$$N(s, a) \leftarrow N(s, a) + 1$$ $$W(s, a) \leftarrow W(s, a) + v$$ $$Q(s, a) \leftarrow \frac{W(s, a)}{N(s, a)}$$where $W(s, a)$ is the total value accumulated through that edge. Each backup refines the mean value estimate $Q(s,a)$, and the visit count $N(s,a)$ grows, gradually reducing the exploration bonus in the PUCT formula for frequently visited actions.
Phase 4: Play
After all simulations complete, select the move at the root proportional to visit counts:
$$\pi(s, a) \propto N(s, a)^{1/\tau}$$where $\tau$ is a temperature parameter. With $\tau \to 0$, this becomes greedy—pick the most-visited action. During training, $\tau = 1$ for the first 30 moves (encouraging exploration of diverse openings) and $\tau \to 0$ thereafter (exploiting the search results).
Notice how PUCT connects back to the bandit theory from Lectures 10–11. The exploration bonus plays the same role as UCB's confidence interval, but PUCT adds the neural network's prior $P(s,a)$ to bias exploration toward moves the network considers promising—a form of informed exploration rather than uniform exploration.
- Initialize root node $s_{\text{root}}$ with neural network evaluation $f_\theta(s_{\text{root}}) = (\mathbf{p}, v)$.
- for simulation $= 1, 2, \ldots, N_{\text{sim}}$ do
- [Select] Starting from $s_{\text{root}}$, traverse tree by selecting $a = \arg\max\!\left[Q(s,a) + U(s,a)\right]$
where $U(s,a) = c_{\text{puct}} \cdot P(s,a) \cdot \frac{\sqrt{\sum_b N(s,b)}}{1 + N(s,a)}$. - [Expand & Evaluate] At leaf node $s_L$, compute $(\mathbf{p}, v) = f_\theta(s_L)$. Store $P(s_L, a) = p_a$ for all legal $a$.
- [Backup] For each edge $(s, a)$ on the path from $s_L$ to $s_{\text{root}}$:
$N(s,a) \leftarrow N(s,a) + 1$, $W(s,a) \leftarrow W(s,a) + v$, $Q(s,a) \leftarrow W(s,a) / N(s,a)$. - end for
- Return $\pi(s_{\text{root}}, a) \propto N(s_{\text{root}}, a)^{1/\tau}$.
Self-Play Training
AlphaGo Zero does not learn from human expert games. Instead, it generates its own training data through self-play, creating a virtuous cycle between the neural network and MCTS that bootstraps intelligence from scratch.
The self-play loop proceeds as follows:
Step 1: Play a game. At each position $s_t$, run MCTS (using the current network $f_\theta$) to produce a search policy $\pi_t$. Sample an action $a_t \sim \pi_t$. Advance to the next state. Repeat until the game ends, yielding an outcome $z \in \{-1, +1\}$ (loss or win).
Step 2: Store training data. Each position generates a training example $(s_t, \pi_t, z)$—the board state, the MCTS-improved policy, and the eventual game outcome.
Step 3: Train the network. Update $f_\theta$ to minimize the combined loss:
where $z$ is the game outcome, $v_\theta(s)$ is the predicted value, $\boldsymbol{\pi}$ is the MCTS search policy, $\mathbf{p}_\theta(s)$ is the network's prior policy, and $c$ is a regularization coefficient. The value loss $(z - v_\theta)^2$ teaches the network to predict who wins. The policy loss $-\boldsymbol{\pi}^\top \log \mathbf{p}_\theta$ teaches it to match the refined MCTS policy. The regularization $c\|\theta\|^2$ prevents overfitting.
Step 4: Repeat. Use the updated network for the next round of self-play. Each iteration produces a strictly stronger player.
MCTS as a policy improvement operator. The relationship between the neural network and MCTS is a form of generalized policy iteration—the same evaluate-improve cycle seen throughout this course. The neural network provides the current policy $\mathbf{p}_\theta$ and value estimate $v_\theta$ (policy evaluation). MCTS uses these to compute an improved policy $\boldsymbol{\pi}$ by looking ahead (policy improvement). Training the network on $\boldsymbol{\pi}$ closes the loop. This bootstrapping effect means the system improves without any external signal beyond the rules of the game.
Self-play has a subtler benefit too. Because the agent always plays itself, the opponent is always well-matched—roughly a 50% win rate at every stage. Against a fixed strong opponent, early games would be all losses: sparse, uninformative feedback. Self-play acts as automatic curriculum learning, keeping the reward signal dense throughout training.
From AlphaGo to AlphaGo Zero to AlphaZero
One of the most remarkable aspects of this line of work is how much simpler each generation became. Each version achieved stronger results with fewer components and less human knowledge. Let's trace that progression.
AlphaGo (2016)
The original system that defeated Lee Sedol 4–1 in a historic match. AlphaGo combined four components:
- A supervised learning policy network trained on millions of human expert games
- A reinforcement learning policy network fine-tuned via self-play against the SL policy
- A separate value network trained to predict game outcomes from positions
- MCTS combining both networks with random rollouts for leaf evaluation
This was a remarkable achievement, but the system depended heavily on human knowledge: the SL policy network needed 30 million positions from human games, and handcrafted features supplemented the raw board representation.
AlphaGo Zero (2017)
AlphaGo Zero eliminated all human data and achieved dramatically stronger play. The key simplifications:
- Single neural network with dual heads (policy + value) instead of separate networks
- No human expert games—trained entirely from self-play starting from random play
- No handcrafted features—raw board position as input (stone positions only)
- No random rollouts—neural network value estimate replaces rollout evaluation
- Residual architecture—deeper, more expressive network with skip connections
The power of eliminating human data. AlphaGo Zero surpassed the original AlphaGo (which used human expert data) within 36 hours of training, ultimately achieving a 100–0 record against the version that defeated Lee Sedol. This demonstrates that human expert data can actually be a liability: it constrains the system to human-like play rather than allowing it to discover potentially superior strategies. AlphaGo Zero discovered novel joseki (standard patterns) and strategies that human experts found revelatory.
AlphaZero (2018)
AlphaZero generalized beyond Go to chess and shogi with minimal game-specific modifications. The same algorithm and architecture achieved superhuman play in all three games within hours of training—defeating Stockfish (chess), Elmo (shogi), and AlphaGo Zero (Go). The only game-specific components were the rules for legal moves and the board representation.
Evaluation and Ablations
Which design choices actually matter? The ablation studies from Silver et al. (2017) give us a clear answer. Three experiments are particularly revealing.
Impact of Architecture
Comparing four network architectures by Elo rating after the same amount of training:
- dual-res (dual-head residual network): ~4300 Elo—the best architecture
- sep-res (separate policy/value residual networks): ~3800 Elo
- dual-conv (dual-head convolutional network): ~3750 Elo
- sep-conv (separate policy/value convolutional networks): ~3100 Elo
Two factors contribute to the dual-res advantage. First, residual connections enable deeper networks that learn more complex positional features. Second, the dual-head architecture forces shared representations between policy and value, acting as a regularizer—the policy head helps the value head generalize, and vice versa.
Impact of MCTS
Comparing Elo ratings with and without MCTS reveals the enormous value of search:
- Raw network (no search): ~3000 Elo
- AlphaGo Zero (with MCTS): ~5000 Elo
- Previous versions: AlphaGo Master ~4800, AlphaGo Lee ~3700, AlphaGo Fan ~3100
- Traditional programs: Crazy Stone ~2000, Pachi ~1200, GnuGo ~400
Search still matters. Even with a neural network strong enough to play at ~3000 Elo from raw policy outputs alone, MCTS adds ~2000 Elo points of playing strength. This underscores a fundamental lesson: combining learned intuition (the network) with deliberate planning (tree search) produces results far beyond either alone. This mirrors the human cognitive process of using intuition to guide search rather than replacing it.
Training Progress
AlphaGo Zero with 40 residual blocks surpasses AlphaGo Lee (the version that defeated Lee Sedol) within approximately 3 days of training. It surpasses AlphaGo Master within ~21 days, and reaches ~5000 Elo by 40 days—all without a single human game as input.
Beyond Games — AlphaZero's Legacy
What happens when you take these ideas beyond board games? It turns out that many optimization and discovery problems can be reframed as sequential decision-making over large combinatorial spaces—and the AlphaZero recipe works surprisingly well:
- AlphaTensor (2022): Discovered faster matrix multiplication algorithms by framing the problem as a single-player game where moves correspond to tensor decomposition steps. Found algorithms faster than any known human-designed method for certain matrix sizes—the first improvement in decades for some cases.
- AlphaDev (2023): Discovered faster sorting algorithms by treating assembly instruction selection as a game. The resulting algorithms were incorporated into standard C++ libraries used by millions of developers worldwide.
RL as search. The unifying insight connecting AlphaGo to AlphaTensor and AlphaDev is that RL can vastly speed up hard optimization problems by representing them as search over combinatorial spaces. The neural network learns a heuristic that guides the search, while MCTS ensures systematic exploration. This paradigm shift—from hand-engineering solutions to learning to search for them—may be one of the most impactful legacies of the AlphaGo line of work.
UCT and the Bandit Connection
Let's make the bandit connection explicit. Recall from Lecture 13 that UCT applies UCB1 at every tree node:
$$a = \arg\max_a \left[ Q(s, a) + c \sqrt{\frac{\ln N(s)}{N(s, a)}} \right]$$PUCT modifies this by weighting the exploration bonus with the neural network's prior $P(s,a)$. But why is there an exploration/exploitation problem during simulated episodes? Because computation is finite. We can only run so many simulations, and we want to allocate them efficiently. UCB minimizes regret in bandits; UCT minimizes "regret" in the allocation of simulation budget. This is the bridge between Lectures 10–11 (exploration theory) and Lectures 13–14 (planning).
MCTS excels when state spaces are large, a simulator is available, and we want an anytime algorithm. It can struggle when horizons and action spaces are both huge—the tree may not grow deep enough to capture long-term consequences.
Summary
We've traced the arc from Go as an AI grand challenge to the general-purpose AlphaZero framework. The progression is striking: each generation got simpler and stronger.
The core recipe: self-play for data generation (with automatic curriculum learning), MCTS for strategic computation guided by learned priors, and a dual-headed neural network providing both intuition and evaluation. The network-MCTS loop is generalized policy iteration—the same evaluate-improve cycle we've seen throughout this course, from dynamic programming to policy gradients.
Beyond games, the same framework discovered faster matrix multiplication (AlphaTensor) and sorting algorithms (AlphaDev). The unifying lesson: RL can vastly speed up hard optimization by learning to search combinatorial spaces, replacing hand-engineering with learned heuristics at every level.