Inference costs 12x training: the new COGS problem for AI teams

The core shift. Inference flops now vastly exceed training flops. DeepSeek's published data shows 6x more daily inference compute than the training run for V3, and François estimates that over a two-year model lifecycle the ratio hits 12x. As AI usage scales, this gap widens. Engineering teams that treat inference as a secondary concern are already behind.

The cost dimension. DeepSeek's reported 80% gross margins come directly from inference innovation—not model quality. For teams running AI in production, inference is the new COGS. The speaker’s comparison to SaaS companies that ignored cloud costs is deliberate: inference cost modeling and optimization must become standard practice, not an afterthought. Open models now offer up to 10x cost reduction over closed APIs, but only if served efficiently.

Where the optimization happens. Dan Fu from Together AI outlined three innovation buckets: model architecture (MoE, SSMs, MLA), kernel and runtime (FlashAttention, PageAttention, custom CUDA kernels), and harness (routing, speculative decoding, compaction). The biggest near-term gains come from better kernel utilization: decode-stage MFU can be as low as 5%, compared to 50% during training. Expert parallelism on 72-GPU GB200 racks is a key hardware lever. Teams should watch for disaggregated prefill and decode on separate GPU pools.

The quality trap. Jeff from Ollama highlighted an underappreciated problem: for open models, the gap between the worst and best inference provider is larger than the gap between open and closed models. Tool calling, quantization precision, and sampling techniques vary wildly. Teams routing across providers need continuous quality monitoring—distribution tests, log-prob drift detection, and agentic eval loops. The implication: cheap inference is worthless if it degrades application behavior.

The compiler approach. Joe from Luminal pitched equality-saturation compilers as an alternative to hand-tuned kernel libraries. Their compiler expands all operations into 14 primitive ops, then searches through rewrite rules to find the fastest mapping for each hardware backend. The key tradeoff: compile time can be hours, but for a model served millions of times, that pays off. This approach commoditizes inference across GPU types but remains niche; most teams should rely on established inference engines like vLLM or SGLang rather than building their own.

What engineers should watch. SGLang's Leon Shaw demonstrated how unified radix trees and hierarchical KV caches (GPU → host → SSD) can reduce TTFT by 56% for long-context agent workloads. The agent workload pattern—shared prefixes, interleaved requests, long-lived sessions—demands caching-aware scheduling and memory compaction. Teams building agent systems should evaluate serving engines that explicitly handle these patterns. The broader signal is that inference optimization is now a full-stack discipline spanning architecture, kernels, scheduling, and hardware allocation.