The AI Cost Crisis: Measure, Right-Size, and Stop the Bleeding

The starting point. Before any optimization, most teams don't know where their tokens go. The speaker reports that 30-40% of new customers arrive with a single shared API key for the whole company. Without per-department, per-use-case measurement, every other decision is guesswork. The first actionable step is to issue separate keys and track spend at the team level.

The right-sizing pattern. The most productive architecture the speaker sees is splitting planning and implementation: the best frontier model handles design and review, while a faster, cheaper model (like Gemini 5.3 Flash) does the heavy token work. One five-person team went from burning $150k/month on Opus to $10k/month on a dedicated Gemma 5.3 instance with no quality drop. This pattern exploits the fact that implementation is token-heavy but quality-tolerant.

The speed-cost trap. Many teams overpay for latency they don't need. Running a model at 120 tokens per second costs roughly 10x per token compared to batching 16 requests at half the speed per request. The speaker advises dedicating high-speed throughput only to interactive workloads. Background agents, CI/CD pipelines, and overnight jobs should run at 20-30 tokens per second. Moving from a burst-to-public-cloud model to dedicated capacity for base load can drop per-million-token costs to under $0.03.

The organizational tax. The talk warns of the slop grenade: generating large volumes of AI output that shift the verification burden to humans. In one example, a team's agent rewrote a quarter of the codebase, then a reviewer used another agent to reimplement the changes from scratch. The output was never used. The root cause is process, not model quality. Teams must own what they generate, and managers should measure outcomes—not token volume.

The bottom line. Measurement is the only prerequisite for meaningful cost reduction. Once teams know their spend per workload, they can right-size models, align incentives (speed over cost), and route traffic appropriately. The speaker's six questions—who spends what, what can be tested cheaper, what is the base load, what is the default model per team, which workloads need speed, how much output is useful—are a concrete starting point for any engineering leader.