The Cheaper Model That Costs More: Why Custom Evals Matter

The proposal. Model cards and broad benchmarks give a misleading picture of real-world model performance. Rayan Krishnan from Vals AI argues that the "jagged frontier" — inconsistent performance across specific tasks — means organizations cannot rely on aggregate scores to choose models. For example, a cheaper model like Opus 5.5 ($4 input) appears cost-effective on paper but actually costs 14% more than Fable 5.1 on a tax benchmark because it uses more tokens to compensate for lower intelligence.

The cost paradox. Cheaper models can become more expensive in practice. The speaker shows that on an Excel modeling benchmark for LBO tasks, the apparent leader falls off the Pareto frontier when you isolate that subcategory. Teams that buy cheaper models based on input/output token costs often end up spending more per task while getting worse results. This directly challenges the common assumption that optimizing for token price is a good cost-control strategy.

The internal case study. Krishnan’s own team of 10-12 engineers spent $1.5 million on AI tokens in a single month when given unlimited access. That is roughly 10x their monthly salary cost. The solution was to build custom evaluations that measured which models actually solved their tasks efficiently. By using these evals to right-size model usage, the team cut token spend by 90% while maintaining output. This is a concrete, if extreme, example of the ROI of eval infrastructure.

The eval framework. To build useful evals, the speaker emphasizes three properties: validity (do the tests reflect real work?), reliability (enough samples for consistent results), and discriminability (can the evals distinguish between models?). He also warns against reward hacking — agents that call other models or give multiple answers to increase scores. Evaluations must be multi-turn and sandboxed properly. The source of data matters: companies should mine their own history of work (PRs, emails, SharePoint) to create relevant tasks.

The competitive advantage. Krishnan claims that most organizations have institutional knowledge locked in employees' heads or legacy systems. Custom evals allow codifying that knowledge into reproducible tests. Once built, evals enable right-sizing, negotiation with vendors, intelligent routing between models, and fine-tuning. This transforms AI cost management from guesswork into engineering discipline.

The vendor angle. This talk is a product pitch for Vals AI's evaluation services. While the data and case study are compelling, the speaker sells the solution. Engineering leaders should take the message — build internal evals — but beware of over-relying on a third-party benchmark provider whose incentives are to sell more evaluations. The operational challenge of curating work-history datasets is undersold.