From Attention to Agents

Where does the optimization happen: in the weights, or in the context?

Twelve chapters that take you from the late-2019 CNN era to the reasoning agents of September 2026, written for readers who last looked closely when attention was first arriving in NLP. Each chapter covers one lever the field learned to pull, and the last chapter puts them together.

Where you left off

In 2019, every NLP task still had its own architecture. How did one model, pretrained once on raw text, end up replacing all of them?

Scale is a strategy

If a transformer keeps improving as it gets bigger, can you predict how much better it will get, and how should you split a fixed budget between model size and data?

Inside a modern pretraining run

Scaling laws tell you how much compute to spend. Where does that compute actually go in a 2026 pretraining run, from filtering the data to the design of each layer?

Teaching it to be helpful

A pretrained model can only continue a document. How do you turn it into an assistant that follows instructions and answers what you actually asked?

The recipe gets cheaper

RLHF needed a separate reward model and an expensive RL loop. How much of that could be stripped away, and why did the field go back to RL anyway?

Steering without retraining

Fine-tuning changes the weights. How far can you get by leaving the weights frozen and searching for a better prompt instead?

Learning to think

Asking a model to show its work made it more accurate. What happens when you train it to reason and let it spend more compute thinking before it answers?

Reinforcement learning grows up

RL on answers you can check automatically is what made reasoning models work. What went wrong when labs scaled it up, and how did they fix it?

Paying for thought

A reasoning model can write thousands of tokens before it answers. What does it take to serve that to millions of people at a price anyone will pay?

Learning to act

A model that reasons still only produces text. How does it learn to call tools and carry a long task to the end without a person stepping in?

Measuring what we built

Most benchmarks saturate within a couple of years of launch. How do you measure what a frontier model can do, and how do you check that it is safe?

Computational cognition

Each chapter turned one lever. What does the whole stack look like with every lever in place, and what does that mean for building a real tool on top of it?

A learning artifact. Every claim, date, and figure is cited inline in its chapter to a primary source (arXiv / ACL / NeurIPS / official report). Built by Owen Wang · more projects.