Your Agent’s Real Benchmark Isn’t Public — It’s Your Production Trace

The real shift is from using production traces only for debugging to turning them into offline simulation benchmarks. Most teams rely on public benchmarks that don’t reflect their tools, domains, or policies. A private benchmark that mimics production is the only way to ship agents with confidence across cost, latency, and success rate.

Building a benchmark isn’t a data collection task; it’s an engineering discipline. The environment must mirror production services, databases, and even simulated users, while verifiers must check final state, trace quality, and artifacts — not just output. Oracle solutions confirm tasks are solvable. This treat-benchmark-as-software approach requires CI, difficulty tagging, and regression gates.

The anti-pattern is fixing every failure in the prompt. Simulations let you test the full stack — model, skills, tools, context — so fixes can land in the right layer (structured output, skill definition) rather than bloating prompts. The process becomes: establish baseline, change one thing, re-run benchmark, then release.

The organizational loop: observability traces feed failures into benchmark expansion, simulation runs test new configs, and a release gate ensures no regression. Benchmarks can be hacked by agents or flawed verifiers, demanding constant curation. Leaders must budget for benchmark development as a product, not a side task.