Speculative decoding is the cheapest speed-up you haven't measured

The speed-up. Speculative decoding is one of the few inference tricks that sounds too good but mostly delivers. A small draft model proposes several tokens, and the target model verifies them in one forward pass, accepting or rejecting the sequence. Because decoding is memory-bandwidth bound, this can roughly triple throughput on local models without changing output quality. Llama app now silently bundles drafters for models like Gemma 4, so users get the gain without configuration.

The vLLM work. The vLLM ecosystem has turned speculative decoding into a trainable infrastructure component. The Red Hat team behind the speculators library showed results across model families: drafting models trained on data generated by a larger sibling (Qwen3 235B data for a 30B target) can match or beat training on the target's own data. Hidden states can be quantized to 4 bits without hurting acceptance, and the Muon optimizer outperforms AdamW for small draft models.

The trade-offs. Speedups are workload-dependent. Dflash-style parallel drafters shine at low concurrency, while an MTP head built into Gemma 4 holds up better under many concurrent requests. Out-of-domain prompts cause acceptance rates to collapse, because a 77M-parameter drafter cannot generalize like the full model. vLLM's adaptive verification reduces waste by deciding how many tokens to verify per request, but teams still need to tune block size and watch memory overhead.

The Gemma design. Martin's visual walkthrough of Gemma 4's MTP drafter is the clearest explanation in the session. The 77M-parameter draft model mirrors the target architecture, projects activations down to 256 dimensions, and cross-attends to the target's KV cache. It also clusters the vocabulary to shrink the LM head from hundreds of thousands of tokens to a few thousand. A default of four drafting tokens is sensible; more than eight usually wastes compute because later tokens are rejected.

The blind spots. The benchmarks are real but narrow. Most measurements come from the serving stack of the people presenting, not from an independent production deployment. The speakers admit that high-entropy creative tasks produce smaller gains, that kernel differences can change outputs, and that batch size affects results. That leaves the central engineering question open: what will this do on your codebase, with your concurrency, on your hardware?

The takeaway. Speculative decoding is infrastructure tuning, not a model upgrade. The safe move is to enable the bundled drafter, benchmark acceptance length and throughput on real prompts, and disable or adjust it when out-of-domain traffic erodes the gain. The technique belongs in the standard evaluation for any self-hosted open model, alongside quantization and serving configuration.