Why Agent Governance Needs Brakes, Not Walls

The problem. Loan software Flexcube sent $900M instead of $8M because a wash account wasn't checked. Human operators overrode a warning, and lenders kept the money. This isn't a banking story; it's what happens when controls fail in systems where humans and software have autonomy. As AI agents get deployed in enterprises, the same failure mode repeats: agents act, humans approve reflexively, and the unexpected becomes inevitable.

The analogy. John Feminella, AI architect at Two Sigma, argues that agent governance should not be about locking everything down. Safety and innovation are not opposites. Just as we give humans boundaries and visibility, we need controls for agents that enable speed, not just prevent disaster. Controls must be composable, not a giant gate that blocks all progress.

Five controls. First, an agent registry makes every agent visible from day one—its tools, endpoints, owner. Without this, you can't govern. Second, risk scoring routes human review proportionally: a ticket summarizer doesn't wait in the same queue as a production deployer. Third, guardrails based on declared purpose: a research agent gets web search, a release agent does not. Fourth, shared evals and metrics so teams don't reinvent evaluation. Fifth, runtime governance with classifiers that escalate only ambiguous or dangerous actions, preventing review fatigue.

The trade-off. The tempting approach is to require heavy approval for everything. That kills innovation and pushes teams into shadow IT. The better approach is to differentiate between open-ended agents (Claude Code) and purpose-built ones, and apply controls accordingly. The talk warns against both sandboxing everything (safe but useless) and letting agents roam free (powerful but dangerous).

What's missing. The framework is sound but assumes most enterprise agents will be purpose-built. In reality, many teams use open-ended coding agents extensively, and the talk's examples of guardrails don't fully address how to constrain a general-purpose agent without breaking its utility. Also, the classifier-based runtime governance is elegant in theory but requires careful tuning to avoid false positives that frustrate teams.

The bottom line. Controls are like brakes on a car: they let you go fast and stop safely. Engineering leaders should adopt this layered approach immediately—registry, scoring, purpose-based tools, shared evals, and runtime classification—before the $900M mistake repeats in their own agent deployments.