AI Dev 26 x SF | Ara Khan: Evals Are Broken Use Them Anyway

The talk argues that both extremes—trusting leaderboard numbers and relying on “vibes”—mislead teams about model quality. Instead, treat evals as decision-support, not truth. Generic benchmarks (ELO, SWE-bench) are saturated and easily gamed; lab-published numbers often reflect benchmark optimization, not real-world behavior.

What matters is task-specific, agentic evals that mirror your workflows. For coding agents, TerminalBench-style tests run full multi-step sessions in isolated containers and grade outcomes with real unit tests. This surfaces practical metrics—turns, tool calls, tokens, wall time, and pass/fail—so you can quantify tradeoffs among quality, latency, and spend. Parallel, containerized runs (e.g., via Harbor/Modal) make this feasible operationally and reduce interference across tasks.

Crucially, evals test three things: the model, your agent harness (tooling, prompts, retries, timeouts), and the task framing. Many failures are harness issues, not model defects. Teams often discover they can beat a competitor’s results on the same model by tuning scaffolding and resource limits, not by swapping models. Cheap models may clear thresholds when the harness is disciplined.

Adoption guidance: stay current but avoid first-mover churn; wait weeks post-release before standardizing. Pick or build evals that map to your domain (not toy algorithms). Track total cost and latency, then set gates for model updates. Beware overfitting to the eval suite—use holdouts and “vibe checks” with real workflows to avoid cargo-cult improvements.

Caveats: claims of outscoring Claude Code are anecdotal and context-bound. The approach demands infra investment, careful containerization, and ongoing maintenance of tasks. But for teams shipping agentic systems, this is the pragmatic path that changes architecture and budgeting decisions.