Your AI Benchmarks Are Useless Without a Cost Axis

Noam Brown says standard benchmark grids are broken because modern models improve greatly with more thinking time or compute, and the plateau is too distant to measure. Without an x-axis for cost, tokens, or time, a higher score could just come from more inference budget, and a better-looking model may be much more compute-efficient.

This distortion misleads both capability assessments and safety governance. Preparedness frameworks from the GPT-3 era don't account for test-time compute scaling. A model might pass low-budget dangerous-capability evaluations yet become harmful when an adversary spends more on inference.

The current release cycle (every 2–3 months) means nobody fully tests the ceiling before the next model arrives, leaving latent capabilities untested. The Erdos conjecture example showed that with a large enough budget ($10K–$100K of compute), earlier models could have achieved breakthroughs that weren't discovered until later.

Engineering leaders should stop relying on static benchmarks. Instead, define per-task budgets (latency, cost, tokens) and evaluate performance curves. This changes procurement, architecture, and safety governance. Beware routing or multi-agent claims that only look better due to more compute. Demand vendors publish curves, not isolated numbers, and build internal evals controlling for inference spend.