Stop Shipping Agents on Vibes: Build a Regression Loop

The pattern. Xander Song, an engineer at Arize AI, presents Harbor, an open-source framework for evaluating agents like integration tests. A team defines a task as a markdown instruction, puts realistic data in a Docker environment, lets an agent run, then executes verifier scripts that inspect the side effects left in the container. The verifier writes a reward, usually pass/fail. Hold tasks and verifiers constant while changing the agent, and the pipeline turns into a regression test.

Why it works. What gets tested matters. Song's team evaluates the hybrid system: Claude Code with their CLI, their MCP server, or the server-side Pixie agent, all pointed at a containerized Phoenix server with a SQLite file. They added a read-only SQL tool because agents were paginating through REST and GraphQL to answer aggregate queries. One prompt line changed to prefer SQL for counts and groups. Harbor showed SQL calls up, GraphQL down, tool calls and turns down, rewards flat.

The tradeoff. Verifiers are the core and the weak spot. Pure verifiers treat the agent as a black box and check outcomes, which makes refactoring safe. Harbor also allows trajectory-coupled checks, and Song uses one to confirm the agent chose SQL over GraphQL. Useful for optimization, brittle for correctness. Rename a skill or tool, and the test fails even when the agent still solves the task. That is the classic implementation-detail coupling problem.

The evidence gap. This is a vendor-aligned case study, not a benchmark. The team is small, regression testing is early, and the data set was generated by Claude with human spot checks. Oracles validate the pipeline before tokens are spent, but only prove the verifiers trust a known solution. No cost, latency, or coverage numbers are shared. The honest admission: realistic data is the hard part, and getting it is more work than the framework.

Adoption limits. Harbor's contract is simple: put the right scripts in the Docker image, and the harness runs them. That works well when the whole system fits in one container, which is true for Phoenix's single-node architecture. Teams with distributed dependencies, S3, queues, or external services will have to carve out stable interfaces or mock external calls, which drags back the mocking pain Harbor tries to remove. Song does not pretend to have solved that.

What leaders should watch. The trend matters more than the tool. Agent evaluation is moving from demos to CI-style gates, and Harbor is a credible early expression of that shift. Engineering leaders should check whether their teams can answer a simple question: for any prompt, skill, or tool change, what evidence proves the agent got better? If the answer is manual testing, they are the target audience.