Benchmarks Are Lying: The Real Cost of Benchmaxxing

Benchmaxing is an incentive problem. Labs train toward what leaderboards measure, not what users need. The result is an industry where popular benchmarks like LMArena are openly gamed, model cards omit contamination, and even Karpathy says rankings don't match real-world quality. Leaders who trust these numbers will make flawed purchasing decisions.

The core problem is cost. Building agentic coding benchmarks with expert-curated tasks costs millions, so labs cut corners: synthetic data, cheap labor, and hard-coded string matches. Contamination becomes the default, not the exception. Heiner's evidence: Opus verbatim reproduced SWE-bench Verified content, yet the model card still cites the score without disclosure.

The solution is not better automated eval. Heiner argues for expensive human judgment: professional writers for Hemingway Bench, domain experts for industry tasks, aligned verifiers, and private holdouts. This creates a tradeoff for engineering teams: high-quality evaluation is slow and costly, but cheap benchmarks actively mislead model selection and agent adoption.

The meta-point: benchmark saturation is often a signal that tasks are broken, not that models peaked. Labs stop publishing evals when 20% of cases fail. Engineering leaders should treat leaderboards as marketing collateral and invest in their own evaluation pipelines, anchored to the specific workflows they want to automate.