Recap — What Is Value Alignment?

How do we build AI agents that do what we really want? Part I introduced three interpretations: align to the user's intent (what they ask for), their preferences (what they'd choose on reflection), or their best interest (what's genuinely good for them).

These can pull in very different directions. A sycophantic chatbot satisfies stated preferences but undermines genuine interests by reinforcing poor reasoning. An agentic AI might faithfully execute your intent while producing outcomes you never actually wanted.

But Part I's entire framework was built around a single user. Part II begins where that framing breaks down: what about everyone else?

Aligning to Social Value or Morality

Part II introduces a fourth interpretation: an AI agent is value-aligned if it does what is morally right. This shifts from "what I want" to "what we want"—where "we" includes everyone affected by the system's actions.

Consider the paperclip maximizer: an AI tasked with manufacturing as many paperclips as possible, which proceeds to convert all available matter—including human beings—into paperclips. This agent is misaligned not just because it ignores the user's broader desires, but because destroying the world is bad for everyone. Even perfect individual alignment doesn't guarantee social alignment.

Does this mean the user-centric framework from Part I was a waste of time? Not at all. Most requests people make of AI are perfectly benign—help me write this email, debug this code. For these everyday cases, user-centric alignment is exactly right. The moral framework operates as a larger ethical context within which user preferences are respected, not as a replacement for them.

Top-Down Moral Alignment

One natural approach: explicitly formulate moral principles and encode them into the system. This is the top-down approach—designers identify the rules, then implement them as hard constraints, reward penalties, output filters, or constitutional rules. The agent follows principles designed by its creators rather than learning values from data.

The immediate challenge: which moral principles? This is an open problem in moral philosophy itself—one that has occupied ethicists for millennia without consensus. Two major frameworks illustrate the difficulty.

Utilitarianism holds that the right action maximizes total net happiness across all affected individuals. It has the appeal of a clean optimization objective—maximize a single scalar—which maps naturally onto RL's reward-maximization framework.

But utilitarianism has well-known problems. It ignores the distribution of happiness: a policy that makes one person miserable while slightly increasing happiness for a billion others might maximize total utility but violate deep intuitions about fairness. And it can override rights—justifying harm to an individual for the "greater good" in ways most people find repugnant.

Common-sense pluralism takes the opposite approach: maintain many moral rules—"Don't lie," "Don't steal," "Don't hurt people," "Keep your promises." This better captures ordinary moral reasoning, which is rarely governed by a single calculus. But rules conflict—what happens when keeping a promise requires telling a lie?—and they require nuanced exceptions that are enormously difficult to enumerate in advance.

Here's the connection to the rest of this course: moral reward hacking. We've seen RL agents find unexpected exploits in misspecified reward functions—a boat-racing agent that earns more points driving in circles than finishing the race. The same phenomenon applies to moral specifications. A utilitarian AI told to "maximize total happiness" might manipulate people's beliefs about their happiness rather than actually improving their lives—the moral analog of reward hacking.

Case Study — Top-Down Agentic AI

Let's make this concrete by returning to the concert ticket agent. How would we apply top-down moral alignment? Two design strategies present themselves.

The first is hard constraints: explicit rules that the agent must never violate. For example, "do not manipulate or mislead other agents," "do not impersonate human buyers," or "never bid more than twice face value." These are analogous to safety constraints in constrained MDPs—the agent optimizes within a feasible set defined by inviolable boundaries.

The second is global objectives: instead of (or in addition to) constraints, modify the agent's reward function to include terms reflecting social welfare. The agent might be rewarded not only for securing a low price for its user but also penalized for contributing to price inflation across the market, or rewarded for promoting equitable access across users.

Both strategies face serious challenges. Which principles should we enforce—fairness, efficiency, profit, consumer protection? How do we resolve conflicts when agents compete? And reward hacking lurks here too: a constraint like "don't raise prices" might push agents toward other exploitative strategies that technically satisfy the letter of the rule while violating its spirit.

More broadly, top-down approaches struggle with coverage (no rule set can anticipate every situation), conflicts (moral rules often clash, and resolving them requires judgment that's hard to codify), and cultural variation (the rules inevitably reflect their designers' values, not the full diversity of moral perspectives).

Real-World Ethical AI Guidelines

These tensions play out in the real world. Jobin et al. (2019) analyzed 84 published AI ethics guidelines worldwide and found broad convergence on high-level themes—but significant divergence on what they actually mean.

The most commonly cited principles:

But look at who wrote these guidelines. The vast majority came from the United States (21), the European Union (19), the United Kingdom (13), and Japan (4). Africa, Latin America, South and Southeast Asia are dramatically underrepresented.

This raises a fundamental question: whose values are being encoded? If the principles governing AI worldwide are written primarily by organizations in wealthy Western democracies, the resulting "universal" guidelines may reflect a narrow slice of human moral diversity.

Bottom-Up Moral Alignment

If writing down moral principles is so hard, maybe we should take the opposite tack: learn moral behavior from examples. This is the bottom-up approach, and it maps directly onto techniques we've studied—inverse RL, imitation learning, and RLHF.

The appeal is clear. Instead of trying to enumerate "Don't lie, except when..." with all its exceptions, we observe how humans actually navigate moral complexity and generalize from those observations. The agent doesn't need a philosopher's theory of ethics; it just needs enough examples to learn the patterns.

But the bottom-up approach faces its own formidable challenges.

But the bottom-up approach faces three serious challenges:

• Moral disagreement — whose examples should the AI learn from? Different communities hold genuinely different views. Should ChatGPT produce depictions of the Prophet Muhammad? These disagreements are genuinely contested. Training on any set of examples implicitly takes a stance.
• Rare and unforeseen cases — a self-driving car trained on millions of miles might never encounter a catastrophic brake failure at highway speed. Moral edge cases are by definition rare, and there's no rule to fall back on when the agent must extrapolate.
• Representation — feedback data is rarely drawn uniformly from humanity. Majority groups and English speakers are overrepresented. The "moral values" learned may reflect existing biases rather than any fair consensus.

Participatory AI — A Middle Path

Neither approach is sufficient alone. Top-down can't anticipate every moral situation; bottom-up inherits the biases of its training data. Participatory AI offers a third path: expand who gets to shape the system's values, and make that process ongoing.

The core idea: include multiple stakeholders, especially affected non-users—people who don't interact with the AI but are impacted by its decisions. For the ticket agent, that means not just the owner but also other concertgoers, venue operators, artists, and the broader community. Values become living, evolving commitments:

• Contextual: What's right depends on the situation, the community, and the cultural context. A single set of rules cannot be applied uniformly across all settings.
• Contestable: Stakeholders can challenge and debate value choices. There are mechanisms for disagreement, appeal, and revision.
• Revisable: The system can and should be updated over time as understanding evolves, as new stakeholders emerge, and as the consequences of design choices become clearer.

In practice, this looks like community advisory boards, user feedback channels that go beyond thumbs-up/thumbs-down, and public consultations before deploying high-stakes systems. These mechanisms are imperfect—participation is costly, power dynamics persist, consensus may be unattainable—but they represent a fundamentally different orientation toward alignment.

Takeaways for Moral Value Alignment

There is no silver bullet. Moral philosophy itself hasn't solved the problem of determining what is right—we shouldn't expect AI alignment research to solve it either. But alignment can be better or worse, and this lecture offers a practical roadmap.

Start with the easy stuff. Some moral intuitions are near-universal: don't kill, don't steal, don't cause unnecessary suffering. These form a solid baseline. Getting the uncontroversial cases right is a prerequisite for tackling the hard ones.

Then do your best to capture the complexities. Each approach contributes something essential:

• Top-down: Think hard about principles, anticipate conflicts between them, and specify nuanced exceptions. Don't just encode naive utilitarianism and call it done. Engage with the philosophical literature. Stress-test your rules against adversarial scenarios.
• Bottom-up: Get creative with training data. Cover as many rare and edge cases as you can imagine. Actively seek diverse perspectives and underrepresented viewpoints. Be aware that your training distribution shapes your system's moral horizon.
• Participatory: Build ongoing feedback mechanisms. Include affected non-users in the design process. Treat alignment as iterative and revisable, not as a checkbox to mark complete at launch.

Here's the connection to everything else in this course: the reward function is a formalization of "what we want the agent to do." Value alignment asks whether that formalization captures not just one user's preferences, but society's values. Every technique we've studied—reward shaping, RLHF, inverse RL—is a tool in the alignment toolkit. The hard part isn't the optimization; it's specifying what to optimize.