Cerebras CTO: 200 TPS is the new batch mode—ultra-fast inference changes everything

Cerebras CTO Sean Lie announces CS4 is in production with OpenAI, delivering over 4,400 tokens per second on GPT-4—roughly 14x faster than GPU baseline. This is not a lab demo: OpenAI uses this speed internally for incident response and research, and is now selectively exposing it to enterprise customers. The implication is

that ultra-fast inference is creating new workload classes—real-time agent loops, interactive reasoning, and multi-step evaluation—that batch-oriented GPU architectures were not designed for. Lie previews CS5, promising 5,000 TPS on frontier models and up to 10,000 on smaller ones, enabled by a modular Nexus platform designed for multi-generational scaling. The tension is clear:

Cerebras is sold out, with most capacity locked into OpenAI's internal use. Engineering teams building on this capability currently face limited access, not limited performance. Lie argues the industry is shifting from '200 TPS is fast' to '200 TPS is batch mode,' drawing a sharp line between traditional GPU architectures (optimized for

throughput) and SRAM-dense wafer-scale designs (optimized for latency). He criticizes competitors like Groq for only showing benchmarks on 30B models, suggesting memory constraints prevent scaling to frontier-level workloads. The broader signal is that inference architecture decisions now have direct organizational consequences: speed unlocks new product categories, but supply constraints create strategic bottlenecks.