Start with Vibes: The Counterintuitive First Step for Agent Evals

The YouTube Ads team discovered that jumping to large-scale automated evals too early caused calibration chaos. Their counterintuitive lesson: start with unstructured, intuition-based 'vibing' to quickly identify failure patterns. This manual approach lets teams iterate radically on prompts and architecture without being held back by brittle eval scaffolding.

Once failure modes are well-understood, the team then builds comprehensive evals with clear rubrics, human raters, and LLM judges. They stress the importance of raters explaining their judgments—pass/fail decisions alone miss why agents fail. A striking example showed an agent removing a legally required disclaimer despite explicit instructions, a flaw invisible to aggregate metrics.

Agent traces proved essential for spotting reasoning gaps that pass/fail rates overlook. Traditional ML principles still apply: maintain test sets, monitor disagreement between human and LLM raters, and focus on patterns rather than hyperfixating on single stochastic failures.

For engineering leaders, this means investing in eval infrastructure as a first-class delivery concern. Define launch criteria early, train cross-functional teams on rubrics, and treat eval development as an iterative process alongside the agent itself. The real bottleneck is often evaluation discipline, not model capability.