Your inference stack is now writing its own GPU kernels

The transcript pivots on a concrete signal: GLM52 is writing its own GPU kernels inside Baseten's inference stack. This isn't a demo—it's a production loop where a model profiles traces, identifies bottlenecks, writes new kernels, and uploads them. It suggests that optimizing inference is becoming a continuous, model-guided process, not a one-time engineering effort. The

immediate implication is that teams must budget for self-optimizing infrastructure rather than static deployments. The conversation moves to the sheer engineering lift required to support new models. Moving from 'making a token' to a production-ready API involves quantization, training a traffic-specific speculative decoder, and handling race conditions that only appear on certain hardware clusters. The

recurring tension is between fidelity to the original model and speed gains—quantization is lossy, but errors can cancel if layers are chosen mathematically. This is not hype; the team published a 39-page paper demonstrating the technique. On the hardware side, the speakers argue that future GPUs like Rubin are moving toward ASIC-like specialization, shifting inference

engineering from kernel writing to infrastructure orchestration. The bottlenecks are becoming memory bandwidth and inter-node KV cache transfer, not raw compute. Video generation remains a different beast—diffusion models struggle with long sequences due to quadratic attention costs, and autoregressive approaches still lack quality. The tradeoff is stark: pay massive compute for full attention or accept