Token bloat costs millions: the engineering playbook for fixing it

The proposal. Alvin Pane, head of agent intelligence at OutRival, argues that the industry's obsession with maximizing token consumption is misguided. Instead of treating high token usage as a badge of honor, teams should focus on extracting maximum intelligence per token.

Why now. Pane's company processes billions of tokens daily across phone, text, and email tasks. At that scale, every redundant model call compounds into latency and millions in compute costs. The problem is structural: developers bolt on paragraphs of rules over time, afraid to delete anything that might break production.

The measurement trap. Teams often reject cheaper models because they disagree with production baselines. Pane demonstrates that production frequently disagrees with itself—roughly 7% of the time in his AA ceiling test. Demanding perfection from a candidate that the expensive system cannot achieve is a mistake.

The three levers. First, caching: structure prompts to maximize prefix cache rates. Second, shrink prompts using an optimizer like DSPy or Jeepa instead of manual editing. Pane achieved a 51% reduction in instruction tokens. Third, downsize the model: break tasks into components, using cheap models for routing and frontier models only for heavy reasoning.

The validation hurdle. After optimization, teams must prove quality didn't degrade. Pane recommends shadow deployments comparing production-versus-production, candidate-versus-candidate, and candidate-versus-production. The key insight: different answers are not necessarily wrong ones. His HarborBite example showed six improvements for every regression.

The bottom line. Before shipping, calculate confidence intervals around the net gain and check critical categories separately—global averages can hide a failing intent. If the lower confidence bound is above zero, ship and keep measuring. Cache, shrink, right-size, validate.