Stop evaluating models. Start evaluating the risk around them.

The dataset. Two New York Life researchers analyzed 1,368 incidents from the public AI Incident Database and found failures are rising almost vertically in the foundation-model era. Their core claim is that failures have structure. Thirteen sub-dimensions across technical strength, deployment controls, and business context cover almost 98% of incidents, with an average of six failure dimensions per incident. That is the strongest concrete signal: incident reports are not random, and evaluation can be organized around them.

The ranking. Not all failure dimensions carry equal harm. When the authors plot frequency against association with harm, deployment controls such as human-in-the-loop effectiveness and monitoring sit closer to harm than technical design or scale. That flips the common instinct to invest first in model quality. A technically strong system with weak controls can be riskier than a modest model with a solid review loop and clear decision authority.

The evaluation shift. The talk argues evals must test the whole deployment system, not just the model. For agentic systems, that means retrieval correctness, tool failure, human review, permission boundaries, and recovery. Risk decomposes into how often the model fails, whether that failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. The same model in two deployment setups has different risk, so model-only evaluation is incomplete.

The method. They propose failure-driven evaluation, grounded in Peter Wason's 2-4-6 experiment on confirmation bias. Instead of confirming that a system works, define a system claim, try to break it, fix what breaks, and repeat. The build-break-fix loop should map out a failure region and route high-risk use cases only to areas where the system is proven. It is a first-principles approach, not a test suite.

The case. The airline chatbot case shows why model-level evaluation is not enough. A customer received incorrect advice about a discounted fare after a death in the family, acted on it, and a tribunal held the airline responsible. A claim such as advice must match approved policy could have been stress-tested before deployment. But the talk exposes its own limits: no implementation details, no production data from New York Life, and no cost estimate for running this kind of eval.

The takeaway. The conclusion is direct: evaluate the full AI system based on risk, not just one component. That is a governance message for engineering leaders, not a technical trick. It asks teams to define materiality, review deployment controls, and treat the evaluator as part of the system being governed. None of that is easy, and the talk offers no ready-made tooling. The practical use is as a checklist for the next high-risk AI review.