Async Distillation’s Speed Gains Hide a Complexity Trap

AsyncOPD tackles the GPU-idle problem in on-policy distillation by fully decoupling student rollouts from teacher scoring and backprop. This yields 1.5–2.7× throughput gains on math benchmarks without degrading accuracy.

The catch: reverse KL divergence (the go-to for training domain-specific expert models) breaks when policy updates shift top-k token distributions. The proposed Monte Carlo fix requires storing multiple local samples per step, adding inference-engine complexity.

For staleness up to ~16 steps, simpler k-step off-policy methods achieve similar speedups with no caching hacks. The decision hinges on whether your team needs extreme asynchrony and can absorb the engineering cost.

Teams scaling distillation should benchmark off-policy first. Full async becomes a specialized investment, not a default optimization.