Agent evaluations are broken: why you should stop chasing benchmarks

Ben Hylak argues that most agent evaluation methods are still rooted in a bygone chatbot era, where simple fact-checking and static eval sets sufficed. Teams are now deploying agents in finance, healthcare, and defense, but the prevailing discourse around evals fails to keep pace. The core shift is from a 'benchmark maxing' mentality

to a 'floor raising' one. The critical focus should be on preventing the worst outcomes—like data deletion or recommending a competitor—rather than chasing best-case capabilities. Hylak's key tactical insight is that agent evals should look like code tests, not prompt playgrounds. He advocates for offline evals that mimic unit tests or end-to-end

tests. He also introduces three practical lessons: clusters are not issues (they lack temporal tracking), code-mode classifiers scale well for trace analysis, and agents are terrible at anomaly detection—so engineers should pull deterministic signals first and only then deploy agents for investigation. This advice comes with warning labels: the field is changing

too fast for heavy investment in hand-crafted eval sets. Hylak's evidence is largely anecdotal and based on his company's work, not independent benchmarks. The talk is more a heuristic guide than a proven playbook, but the observation that most evals are 'security theater' aligns with what many engineering teams are already experiencing.