LLM inference is a memory problem, not a compute problem

LLM inference cost is a recurring operating expense that scales with usage, unlike training's one-time cost. The core bottleneck is the KV cache—memory grows linearly with context length and concurrently with users, creating a tradeoff between quality, latency, and throughput.

Prefill is compute-bound; decode is memory-bound. Understanding these phases drives optimisation decisions. Model-level techniques like quantization (int8/int4) and grouped-query attention reduce memory footprint, often at negligible quality loss in practice. Flash attention and multi-head latent attention further compress compute.

On the serving side, KV cache management via page attention, continuous batching, and prefix caching dramatically improves throughput. Benchmarks show VLM and SGLang perform similarly for standard workloads, but SGLang can be 3-4× faster for agentic tasks with repeated prompts.

Speculative decoding is overhyped—presenters found it ineffective. The real signal is that architecture decisions (attention mechanism, serving engine) matter more than squeezing last bit of FLOPs. Teams should fix one constraint (latency or throughput) and model the capacity tradeoff systematically.