NVL72 Racks Create a Scheduling Cliff That Kubernetes Cannot See

The core problem. Training jobs spread across hundreds of GPUs and at every step every GPU must exchange data with every other GPU. The job runs at the speed of its slowest connection. On NVL72 hardware the NVLink domain is the entire rack of 72 GPUs. Stay inside the rack and every link runs at 900 GB/s. Cross to another rack and you drop to 50 GB/s over InfiniBand. That is an 18x speed cliff.

What Kubernetes misses. Kubernetes places pods one at a time wherever there is room. Quota tells you how many GPUs you may use but not where they should go. A 512-GPU job can be admitted correctly within quota and still land scattered across racks running at a fraction of peak performance. The speaker shows a 16-GPU example: a single misplaced GPU forces every communication round onto the network instead of only the final round. The same job the same quote usage but dramatically slower.

The scheduling fix. The Q batch scheduler fills the gap. Platform teams declare the cluster topology once as an ordered list of node labels. Users add a single annotation on their pod template. The annotation has three modes: unconstrained (no placement promise), preferred (best effort shape), and required (don't start until the shape is right). The rough rule is that pre-training uses required, post-training uses preferred, and inference or single-node workloads use unconstrained.

Production reality 1: test pods and preemption. When Cohhere first turned on topology-aware scheduling it interacted badly with CoreWeaves HPC verification pods. Q counted the test pods as real usage and blocked customer jobs. The fix was to omit the Q label on test pods set their priority to negative one and let ordinary Kubernetes preemption evict them when a real workload arrived. Q never knew they existed.

Production reality 2: fragmentation. Over time jobs finish in random sizes and leave holes. Aggregate free capacity can look plentiful while no single rack has enough contiguous GPUs for a large job. The cluster is a third empty but the hero workload cannot start. Q can slow the decay with different placement policies but nothing today can bend the curve back up. The only things that open a rack are jobs finishing or preemption.