The Real AI Cost Lever Is Cost Per Task, Not Token Price

The core metric. OpenAI's applied AI team argues that per-token list prices mislead budget decisions. The metric that matters is cost per completed task, including human intervention when it fails. A canceled flight example makes the point: a cheap per-token chatbot could not rebook a passenger, so the real cost included an agent's time. Perplexity reported 9% more accurate results at half the cost with Astra, and Notion reported better accuracy at half the cost per task on migration.

Right-sizing models. Start by defining the task and the accuracy threshold the business expects, then build a representative eval set. Test models, reasoning settings, prompts and harnesses, and plot accuracy against cost. A smaller model often wins: Luna at high or extra-high reasoning beats larger older models, and OpenAI says customers use it for 80 to 90% of workflows. The catch is that this holds only when the eval set reflects production usage.

Four levers. Beyond the model, the panel lists prompt caching, programmatic tool calling, reasoning effort, and batch or flex processing. Prompt caching can cut cached input costs by up to 90%. Programmatic tool calling moves intermediate result processing out of the model and into code; legal software company Clio reported 38% fewer prompt tokens in multi-step document analysis with no quality loss. Batch processing offers roughly 50% lower pricing for a 24-hour window. Each changes latency or determinism; the tradeoffs are real.

Cache discipline. Prompt caching is automatic, but teams rarely design prompts to use it. Keep the start of the prompt stable; newer model families support explicit breakpoints for stable sections and explicit-only caching for frequently changing ones. OpenAI added a caching dashboard and a diagnostics API for finding misses. Blitzy, an AI software company, improved cache reuse from 24% to 90% by moving to Luna's tool calling loop, reporting 8.5x fewer output tokens, 87% cheaper cost versus GPT-5.4 Mini, 2.2x more context.

The trade-offs. Savings depend on organizational discipline. Lower reasoning effort can degrade quality, so evals must run iteratively. Batch saves money only when immediate responses are not needed. Programmatic tool calling keeps context out of the model, but moves judgment into code that must be maintained. All the customer percentages come from OpenAI's own engagements, so they are directional evidence, not independent benchmarks.

The takeaway. The useful signal for engineering leaders is that AI cost is largely a workflow design problem; model selection is only part of it. Instrument cost per completed task, define accuracy thresholds, and treat caching, tool orchestration and reasoning effort as budget levers. Validate any migration with internal evals before trusting vendor-reported savings.