Silent vLLM Bugs: How Stale State and Integer Overflow Hid in Plain

The problem. AI21 engineers debugging vLLM inference encountered two bugs producing silent gibberish—no crashes, errors, or warnings, only bad outputs that looked like quality issues but were engineering failures. These bugs appeared only under load, only with Mamba models, and only after thousands of requests, making them nearly impossible to reproduce without deliberate stress testing.

First bug: stale state. Engineers reduced GPU memory utilization to 20% and flooded vLLM with concurrent requests. Request 8,854 returned gibberish deterministically at temperature zero. Using Hugging Face Transformers as a baseline, they built a log-probability divergence comparison tool to isolate where the output deviated. The root cause: vLLM's scheduler sometimes ran decode before prefill for new requests, reading stale Mamba state from previous requests instead of initializing it. The fix: mark new request tokens as requiring prefill before decode.

Second bug: integer overflow. During RL training, log-probability spikes appeared every 12 steps between rollout and training, with identical inputs producing different outputs. Increasing rollouts per prompt from 8 to 128 made the bug appear immediately. But reducing GPU memory—the obvious lever—made it disappear, which was misleading. The actual cause: Mamba kernels used unsigned 32-bit integer indices for state pointers. When the offset exceeded 4 billion, it wrapped around instead of throwing an error.

The pattern. Both bugs shared the same characteristics: silent failures in Mamba's state cache, surfaced only under memory pressure, and detectable only through log-probability forensic comparison. Neither bug produced a crash, warning, or observable error. Both required the engineers to propagate request IDs through vLLM's internals, thread identity through opaque tensor operations, and inspect metadata at the exact failing moment.

The trade-offs. Debugging at this depth means getting hands-on with the inference engine's scheduler, kernel calls, and memory management—far beyond typical model evaluation workflows. The engineers had to modify vLLM source code to add breakpoint conditions. This level of investment is justified when silent quality degradation affects training signals or production outputs, but it's not scalable across every inference issue.

The takeaway. For teams running stateful inference with Mamba or other recurrent architectures, log-probability comparison tools are essential infrastructure, not nice-to-haves. Reproduce under memory pressure rather than normal conditions. And when failures are silent, thread identity through every layer until you can attach a debugger to the exact failing request.