Evals, cost controls, and review loops now decide agent outcomes
Three talks converge on one point: model IQ matters less than evals, cost, and review loops.
Build the check first. The API choice sorts itself out.
Trust is a check. Autonomy only works when the team can verify the output.
Coinbase stages agents from supervised to sampled to exception-handling only after measured confidence.
Duolingo auto-approves just 10% of PRs, all low-risk, and feeds misclassifications back into an eval set.
The model is interchangeable in both stories. The check is the durable part.
Harness IQ is flat. Coding harnesses no longer add intelligence, but they do set cost and control.
The UC Berkeley study ran seven frontier models across several harnesses and found no score difference on SWE-bench Light, while cost per task varied.
Separately, one builder replaced Claude Sonnet with Llama 3.2B after running her own evals.
That makes cost, permissions, and eval data the real selection criteria.
Comprehension debt grows. Faster agent loops outpace the humans who must review what ships.
IBM calls it comprehension debt: nobody can reason about the system during an outage, and engineers accept output without reading it.
Duolingo keeps the correction loop small, one or two reports a week, by narrowing scope.
Plan the review bottleneck before scaling autonomy.
My take
I'd put money on the check this month.
The Coinbase ladder and Duolingo's feedback loop both treat the eval set as the product, and I think that's right.
What I'm less convinced about is the smoothness of the safety story: no talk here shows a controlled defect-rate comparison, so the claims stay circumstantial.
Start with one narrow, reversible workflow, define what good looks like with the people who do the work, and only then let an agent run.
That is how you get autonomy without comprehension debt.
The videos behind this brief
We watched these so you don't have to.
Agent Autonomy Is a Confidence Ladder, Not a SwitchAI Engineer
Letting AI approve low-risk PRs cut Duolingo's median merge timeInfoQ
Agent Harnesses Stop Adding Coding IQ. Now They Manage Cost and Risk.AI Engineer
Apps Become APIs: The Real Moat Is Interoperability, Not UIInfoQ
Agent loops run by themselves; comprehension debt is the new bottleneckIBM TechnologyVideo and thumbnails remain the property of their respective creators. tldw.news provides editorial analysis, commentary, and discovery links to original content.