Agent Autonomy Is a Confidence Ladder, Not a Switch

The core claim. Evan Kormos, a player-coach at Coinbase, reframes agent rollout as a workflow and trust problem, not a model capability problem. He models policy-driven processes as DAGs with a specific, gated direction, citing Apache Airflow as a familiar example. Inside each node, an agent loops with a human, represented as a directed cyclic graph, where LangGraph manages state. The formula he lands on is "deterministic on the outside, adaptive on the inside."

The operational shift. The reason to formalize this is parallelism. Work that used to be too expensive or complex to coordinate can now run independent reviews side by side, then rejoin the main flow with checks intact. But volume moves. If code review gets faster, design and security scans become bottlenecks. Kormos argues operational visibility into the whole process matters more than the workflow authoring tool, a point most tooling reviews miss.

The check is trust. Agents are probabilistic, so exact output comparison is not enough. He recommends representative datasets, repeated evals, and traces as runtime logs to build a confidence mechanism similar to unit tests for traditional software. "The loop is only as trustworthy as the check" is the sharpest line in the talk. That means defining what good looks like with domain experts before scaling.

The autonomy ladder. The most actionable idea is hill climbing. Day one is supervised: humans verify every output, refine datasets, and confirm thresholds. When measured confidence clears policy, move to sampling, where humans review a slice and traces cover the rest. Finally, humans handle exceptions while still refining the eval sets. This gives teams a concrete way to answer "when can the agent run unsupervised?"

The trade-offs. There is no production data here. The examples, commercial lending and token listing, stay high level. Kormos does not explain how to set a defensible confidence threshold, what sample size is statistically valid, or how to handle adversarial inputs and audit burden. "Quality is a team sport" gets a slide but little detail. This is a governance framework from an experienced practitioner, not a measured result.

What to watch. Engineering leaders should take the staging model as a starting point, not a guarantee. The real work is building the check: an evaluation set, tracing, and a clear exception path. When agents speed up one node, plan for bottlenecks elsewhere. The talk is most useful as a conversation starter for regulated workflows, where the default instinct is to block agents instead of phasing them in.