Your Agent Fix Just Broke Something Else – Agent Regression Testing

The problem. A one-line prompt change to a support agent asks it to confirm before every action. The security engineer approves it. It looks harmless. But in production, the agent now leaves tasks incomplete when users walk away. Safety improves 17 points, task completion drops 17 points. Net zero. The code diff showed nothing wrong. The runtime behavior shifted silently. This is the fundamental challenge of agent regression: agents fail at runtime, not in static code.

Why traces matter. The speaker argues that traces are becoming the new source of truth. A code diff tells you what changed. A trace tells you what actually happened — the tool calls, the sub-agent handoffs, the branching paths. In traditional software, logic is written down before deployment. For agents, logic only exists at runtime. Observability fills the gap between what was written and what actually executes. Without traces, you are shipping on intuition, not data.

The five-step regression loop. The talk outlines a practical process: 1) Select a production data set covering normal workflows, known failures, edge cases, and successful paths you want to protect. 2) Build evaluators — LLM-as-judge for subjective metrics like task completion, code-based evals for deterministic checks. 3) Run a baseline experiment: run the agent against the data set and score it. 4) Make a change and rerun the experiment.

The trade-off. The approach is sound but requires upfront investment. Building a representative eval suite is hard. You need to decide which metrics matter and how to score them. LLM-as-judge introduces its own variance and cost. The speaker is selling Arize's platform, which makes this easier, but the principles are vendor-agnostic. Teams must weigh the cost of building versus the cost of silent regressions.

Closing the loop. The talk introduces two key takeaways. First, every failure should become a test: add failing production traces to your regression suite so the system gets harder to break over time. Second, the next frontier is agents debugging themselves — connecting coding agents to observability data so they can investigate and propose fixes autonomously. That's a speculative leap, but the first takeaway is immediately actionable.

What's missing. The talk does not address how to handle false positives from eval scores, how to manage eval cost at scale, or how to evaluate multi-agent handoffs beyond simple correctness. The demo is polished but shows no real production data over time. Engineering leaders should adopt the loop concept but treat the self-healing agent vision as aspirational, not proven.