Testing AI Agents: Plant the Defect, Then Find It

The bottleneck. Nathanael Weill, who runs AI delivery at Air Canada, starts with the problem most enterprise teams hit: no public benchmark exists, and only busy domain experts can grade the agent. On an earlier telecom RAG project, his team got one expert evaluation per week, and the grading sessions were contradictory. Without a fast loop, the team could not tell whether added reasoning steps helped or broke retrieval. The result was a complex pipeline that was not measurably better.

The inversion. The fix was simple: start with a known document, have an LLM write a question about it, run the agent, and check whether the agent retrieves the same page. Weill did not invent this for RAG, but he turned it into a general principle: plant a known defect or answer, then test whether the agent finds it. That removes expert opinions from the grading loop and makes success checkable in code.

The impact. That shift took his team from one expert evaluation per week to dozens per day. Removing pipeline components mattered as much as adding techniques from papers, and accuracy climbed over weeks. His working rule: if you cannot grade the agent, you cannot improve it.

The contract case. At Air Canada, the agent analyzes supplier contracts: spot clauses, find supporting documents, write amendments, and classify them. The team wanted to diff old and new contracts, but the company did not keep original versions. So they generated tests instead. Using known rules, like a cap above one million dollars, they built 275 contracts with planted violations. Taking signed contracts and deleting clauses was the strongest method, exposing that the agent missed most removed clauses.

Code before judges. Weill's team avoided LLM-as-a-judge whenever possible. The noise from an LLM judge was nearly as hard to control as the agent itself, and tuning the judge consumed effort without a stable signal. Code checks handled consistency: change supplier names, reorder clauses, rerun the same contract, and verify the flag count stays stable.

The real loop. Expert templates with blanks became a generator: fill blanks with values that do or do not break rules, and expected flags are known without model grading. Generated tests are a starting point, not a go/no-go decision. Weill uses expert feedback to improve the tests and prompts, not just the agent. Sometimes the solution must be re-engineered so it can be evaluated, a tradeoff teams should accept.