Learning to act

Chapter 8 taught the model to think cheaply enough to keep the meter running. Now it needs to act. A reasoning model can run a chain of thought all day and still not move a pixel. To affect the world it needs two missing pieces - a way to call out (tools) and a way to listen back (observations). Wire those into a loop and the chatbot becomes an agent. This chapter is that wiring: the ReAct loop as the spine, three answers to "how do tools work" (research, API, open standard), the wild 2023 wave of autonomy, the moment the model got a mouse and keyboard, then the three things frontier labs added in 2025-26 to make agents useful for real work - a training pipeline of executable environments, a measurable notion of long-horizon competence, and the discipline of engineering their context. It closes on the security problem the whole stack inherits when text from the world can steer text into the model.

1.The loop that defines an agent

The cleanest definition of an agent is also the smallest: a model that interleaves thinking, acting, and observing, in a loop, until the task is done. That pattern has a name and a paper. ReAct - "Synergizing Reasoning and Acting in Language Models" - proposed it in October 2022 and the design has barely shifted since. (Yao et al., ICLR 2023.)

The trick is that the "thought" step is just chain-of-thought reused inside a control loop - the same seed planted back in Chapter 3 and trained into the weights in Chapter 6, now placed where it does mechanical work: deciding the next action. The "observation" is whatever the world hands back when you act on it: a search result, a stack trace, a screenshot. The loop closes because each observation feeds the next thought.

The numbers in the paper are worth pausing on, because they justify why this pattern took over. On ALFWorld, a text-based household-task benchmark, ReAct beat imitation- and RL-trained baselines by +34 absolute success-rate points. On WebShop it beat baselines by +10. Both numbers came from one or two in-context examples - no fine-tuning, no new weights. The agency was sitting in the prompt all along; the loop just let it out.

2.Three answers to "how does it call a tool?"

A loop is only as useful as the things the model can do inside it. The story of tool use is a story of three answers - one from research, one from product, one from standards - arriving roughly a year apart.

The research move: teach the model when to call

Toolformer (Meta AI, February 2023) showed a base LM can self-supervisedly learn when an API call would help: insert a candidate call into its own training data, check whether the call reduces the downstream loss, and keep only the helpful ones. Calculator, search, QA, translation, calendar. The model learns to reach for an external tool the way it learned to predict the next word. (Schick et al., NeurIPS 2023.)

The product move: a structured-JSON API

On June 13, 2023 OpenAI shipped function calling on gpt-4-0613 and gpt-3.5-turbo-0613 - a structured JSON name + arguments matching a developer-supplied schema. You describe your function, the model decides when to invoke it and emits clean JSON, your runtime executes it and feeds the result back. Tool use stopped being a prompt-engineering pattern and became a model capability with a fixed wire format. (OpenAI, June 2023.)

// developer registers a function schema
{
  "name": "get_weather",
  "parameters": { "location": "string", "unit": "string" }
}

// model emits, given "what's the weather in Seoul?"
{
  "name": "get_weather",
  "arguments": { "location": "Seoul, KR", "unit": "c" }
}

// developer runs the function; result is fed back as the next message
{ "temp": 14, "sky": "clear" }

The standards move: an open protocol for tools

Eighteen months later, on November 25, 2024, Anthropic introduced the Model Context Protocol (MCP), an open standard (spec version 2024-11-05) for how LLMs connect to tools and data sources. Python and TypeScript SDKs at launch; reference servers for Google Drive, Slack, GitHub, Git, Postgres, and Puppeteer. The point of a protocol - rather than another vendor API - is that every model can talk to every tool without bespoke glue, and the tool describes itself once. (Anthropic, Nov 2024.)

The interesting thing is not the launch, it is what happened next. On March 26, 2025 OpenAI shipped MCP support in its Agents SDK (Sam Altman). Two weeks later Google confirmed the same for Gemini (Demis Hassabis, Apr 9 2025); Microsoft announced native MCP for Windows at Build in May (Windows blog, May 19 2025). The spec kept iterating: 2025-03-26 added OAuth 2.1 and a Streamable HTTP transport, 2025-06-18 added structured tool output and Resource Indicators, 2025-11-25 added durable tasks and OIDC discovery, and the current 2026-07-28 revision replaced the initialization handshake with per-request version negotiation, a mandatory server/discover RPC, and first-class extension points for Tasks, Skills-over-MCP, and inline UI apps (MCP versioning). A first-party MCP Registry preview opened on September 8, 2025, backed by Anthropic, GitHub, PulseMCP, and Microsoft. In under two years an Anthropic-led standard became the wire format every frontier lab agreed to speak.

Notice the through-line. Research asked can a model learn to call APIs?; product asked can we make that a routine capability?; standards asked how do tools describe themselves to any model? Each move was needed; none of them, on their own, makes an agent.

3.The wild 2023 wave

The moment function calling shipped, builders started asking the obvious next thing - what if the model decides its own goals, splits them into sub-tasks, and runs the loop unattended? In spring 2023 AutoGPT wrapped GPT-4 in a self-prompting loop with web search and file I/O, decomposed a user goal, and ran without per-step approval. It became one of the fastest-starred repos in GitHub history. Most of the runs were unproductive - the loops chased themselves, ran up token bills, and got stuck - but the proof-of-concept was loud enough that "autonomous agent" became a category overnight.

The serious research counterpart arrived two months later. Voyager (NVIDIA / Caltech, May 2023) dropped an LLM agent into Minecraft with three pieces: an automatic curriculum that proposed the next task, an ever-growing skill library of executable code the agent wrote and reused, and an iterative prompting loop that fed environment errors and self-verification back into the next attempt. It was a code-writing ReAct agent for a sandbox world, and it learned faster and explored further than every prior LLM-in-Minecraft baseline. (Wang et al., 2023.)

4.The eval shadow grows up

Every chapter has an eval that defined the scoreboard. BBH defined Chapter 3, MMLU defined Chapter 4 - Chapter 9's is SWE-bench: 2,294 real GitHub issues across 12 popular Python repos. The task is unglamorous and exactly right for agents - read the issue, edit the codebase, make the project's hidden tests pass. (Jimenez et al., ICLR 2024.) At publication the best system tested (Claude 2 with a retrieval pipeline) solved just 1.96% of issues. The benchmark was effectively unsolved.

That number didn't stay there. Within a year the field had to admit a separate problem: some SWE-bench problems were underspecified or had flaky test setups, so headline numbers stopped meaning much. On August 13, 2024 OpenAI released SWE-bench Verified: a 500-problem subset, each independently reviewed by three engineers, drawn from a pool of 1,699 candidates, with containerized Docker environments to make runs reproducible. (OpenAI, Aug 2024.) SWE-bench Verified became the headline metric in every frontier-model launch that followed - a low-single-digit ceiling in late 2023 turned into 74.9% at GPT-5's launch (OpenAI, Aug 7 2025), 74.5% at Claude Opus 4.1 (Anthropic, Aug 5 2025), and 76.2% at Gemini 3 Pro's launch three months later (Google, Nov 18 2025). Headline numbers on the Verified subset kept climbing through 2026 and are now less informative than they used to be; Chapter 10 covers the held-out SWE-bench Pro and Terminal-Bench families that took over as the harder scoreboards.

And here the eval-shadow thread bites. A benchmark of self-contained "fix this issue, pass these tests" tasks is a beautiful target - and a partial one. It doesn't capture long-horizon collaboration, code-review pushback, maintainability, or whether the patch would survive a real codebase six months later. Race the metric long enough and you get a model optimized for the metric. The full Goodhart treatment waits for Chapter 10; here it is enough to note the villain showed up wearing a green test bar.

Pass-rate on a fixed task set only tells you so much about an agent, though. METR proposed a different metric in March 2025: the 50%-success time horizon, meaning the length of task (measured by how long an expert human takes) at which the agent succeeds half the time. Under the Time Horizon 1.1 methodology (Jan 2026, 228 tasks, 31 of them over 8 hours), the post-2023 doubling time is 131 days, and the post-2024 slope is faster still at 89 days. That refit put GPT-5 around 214 minutes and Claude Opus 4.5 around 320 minutes at the same lineup's snapshot. (Kwa et al., Mar 2025; METR, Jan 2026.) Since then, METR has been publishing per-launch pre-deployment reports (Claude Opus 5.5 Sep 22 2026, GPT-5.6 Sol Jun 26 2026) rather than a refit of the full curve (METR blog). The number to remember is not the specific horizon on the day you read this. It is that the eval landscape now has a moving axis for exactly the thing agents are supposed to do: keep working. Chapter 10 tables the rest of the 2026 scoreboard.

5.The model gets a mouse and keyboard

Late 2024 collapsed three things into one beat. On October 22, 2024 Anthropic released computer use with the upgraded Claude 3.5 Sonnet - the model takes screenshots as input and emits mouse and keyboard actions, like a person at a desk. The launch numbers: OSWorld screenshot-only at 14.9% (next-best system at 7.8%, rising to 22.0% with more steps); SWE-bench Verified jumping from 33.4% → 49.0% over the previous Claude 3.5 Sonnet; TAU-bench retail 62.6% → 69.2%, airline 36.0% → 46.0%. (Anthropic, Oct 2024.)

OSWorld is the benchmark that justified the capability: 369 real-world Ubuntu (and some Windows) computer tasks, end-to-end, execution-graded. At publication humans cleared 72.36%, the best model managed 12.24%. (Xie et al., NeurIPS 2024.) Then on January 23, 2025 OpenAI shipped Operator, a research preview running on a new model called CUA (Computer-Using Agent) built on GPT-4o, also browser-and-screenshot driven. (OpenAI, Jan 2025.)

A benchmark, an Anthropic release, an OpenAI release - all inside about three months. The pattern of the chapter is clearest here: it isn’t one paper, it’s a cluster, and the cluster keeps moving.

6.The agent the reader already knows

The on-ramp the reader actually has is a coding agent. The category arrived in March 2024 with Devin from Cognition - pitched as the first AI software engineer, end-to-end: plan, write code, run tests, open a PR. Eleven months later, on February 24, 2025, Anthropic announced Claude 3.7 Sonnet, its first "hybrid reasoning" model with an extended-thinking mode, alongside Claude Code, a command-line research preview for agentic coding - delegate a real engineering task from the terminal and let the agent run. (Anthropic, Feb 2025.)

If you're reading this on the site this course is being built with, you already know the shape of the loop. Claude Code is a ReAct agent with file I/O, shell access, MCP tools, and the extended-thinking mode of Chapter 6 dropped in at the "thought" step. The metaphor closes in a slightly recursive way - this chapter is being typed by an agent of the kind it describes.

7.Training the agent, not just prompting it

Everything above assumes the agent's competence comes from the same weights the chatbot uses. Through 2024 that was true - the ReAct trick was that competence sat in the prompt. In 2025 the frontier moved: labs started post-training models specifically to be agents, using the same RLVR lever Chapter 6 introduced but with real repos as the verifier. The trick was building environments where a training run could roll out an agent, execute its code, run the repo's own test suite, and use the pass/fail signal as reward.

The public pipeline arrived quickly. SWE-Gym (Dec 2024) packaged 2,438 Python tasks, each with an executable environment and a test suite, and reported up to +19 absolute points on SWE-bench Verified when used to train an open agent - and 32.0% at inference with a learned verifier. (Pan et al., 2024.) R2E-Gym (Apr 2025) auto-synthesized 8.7K tasks with the SWE-Gen pipeline, dropping the reliance on hand-written issues, and reached 34.4% pass@1 on SWE-bench Verified with a 32B model. (Jain et al., 2025.) SWE-smith (Apr 2025) turned any Python repo into an executable environment with synthetic test-breaking tasks, released 50K instances from 128 repos, and its SWE-agent-LM-32B reached 40.2%. (Yang et al., 2025.) SWE-rebench (Nebius, May 2025) built a continuously refreshed pipeline of 21K+ tasks from 3,400+ repos as Docker images, aimed at both training and decontamination. (Badertdinov et al., 2025.) Berkeley's SkyRL-Agent (Nov 2025) put the full training stack together - rollouts, environment, RL trainer - for multi-turn agents and reported 39.4% at more than 2x cheaper. (SkyRL-Agent, 2025.)

8.Engineering the agent's context

A ReAct loop runs for hours and its context window does not. The engineering problem that ate 2025 was less "which model" and more "which tokens live inside its window at any given step." Andrej Karpathy and Tobi Lutke started calling this context engineering in late June 2025 (Karpathy, Jun 25 2025; Willison, Jun 27). Anthropic wrote the term into their public engineering practice a few months later, framing it as the discipline of curating what actually goes into the limited context window at each step (Anthropic, Sept 29 2025).

Three sub-tools emerged, each with a public reference implementation:

• Compaction. When the conversation nears the context limit, the harness summarizes the older turns on the fly and keeps working. Claude's platform docs describe this as the default long-conversation strategy (Anthropic docs).
• Persistent memory. A file-system-shaped memory tool (public beta Sept 29 2025) that lets the agent read and write files in a scoped /memories namespace across sessions (memory tool). The MemGPT / Letta line (arXiv Oct 2023) is the prior art for the pattern: OS-style paging between a small live window and a larger store (Packer et al., 2023).
• Subagents. Anthropic's June 2025 write-up of its multi-agent research system reported that an orchestrator delegating to parallel workers with independent context windows beat a single-agent baseline by 90.2% at ~15x the tokens (Anthropic, Jun 13 2025). Cognition's Walden Yan pushed back the day before with "Don't build multi-agents," arguing that parallel workers make coherence impossible unless they share the full trace (Cognition, Jun 12 2025). The two are less opposed than they read: Anthropic's win is on embarrassingly-parallel research; Cognition's warning is about single artifacts that demand a single decision-maker.

A ReAct agent with none of these can act on the world. A ReAct agent with all of them can act for a long time on the world - the METR horizon of the previous section is exactly the axis this section optimizes.

9.Prompt injection: the agent's security problem

There is a cost to putting text from the world into a model that treats text as instructions. Simon Willison named it in September 2022 - prompt injection, by analogy with SQL injection: instructions in user-supplied data hijack the model that was supposed to be processing the data (Willison, Sep 12 2022). The agent version is worse. Greshake et al. formalized indirect prompt injection in Feb 2023: the attacker doesn't talk to the model at all; they plant instructions in a document, an email, a webpage, a calendar invite, and wait for the agent to read it while executing on the user's behalf (Greshake et al., 2023).

Two 2025 incidents made the theoretical bite concrete. EchoLeak (CVE-2025-32711, Aim Labs, June 2025) was the first documented zero-click indirect-injection exfiltration of a production agent: a crafted email to Microsoft 365 Copilot silently steered it into reading a user's private files and leaking the content (NVD CVE-2025-32711). "Invitation Is All You Need" (SafeBreach, Black Hat 2025) hid instructions inside a Google Calendar invite, then asked Gemini to "summarize my week" - the summarizer read the invite as instructions and controlled the user's smart-home devices (SafeBreach, 2025). The OWASP LLM Top 10's v2025 release keeps prompt injection at LLM01 (OWASP, 2024).

Two defenses have primary-source teeth. Anthropic's constitutional classifiers (Sharma et al., Jan 2025) run input and output classifiers trained from synthetic data against a written safety spec; 3,000+ hours of red-teaming produced no universal jailbreak, at a 0.38% false refusal rate. Google DeepMind's CaMeL (Debenedetti et al., Mar 2025) takes the confused-deputy framing seriously: a privileged planner emits a program in a restricted DSL, a quarantined LLM handles untrusted content, and a custom interpreter tracks provenance so untrusted data cannot influence the control flow - 77% of AgentDojo tasks solved with a provable security bound versus 84% undefended. Neither is a settled solution. They are the shape the field is exploring.

10.What agency needed that conversation didn't

Pull the chapter back to first principles. A pure chat model needs only enough state to finish the next message. An agent needs four things conversation never asked it to have, and the history above is largely a story of inventing them.

That list is editorial - no single paper to cite - but each item points to a citation earlier in the chapter. The reason agency took until 2024–2025 to feel real is that all four ingredients had to land together. A model that could think (Chapter 6) but couldn't remember, observe, recover, or plan was a clever conversationalist. Wire those in and you get the thing on your laptop.