Fix the data pipeline, not the model: 15% to 90% GPU utilization

The bottleneck. A multimodal SFT pipeline ran at 15% GPU utilization. The cause was not kernels. Before an image reached the Qwen3-VL model, the stack fetched JSONL examples and linked images from S3, resized and normalized with PIL, tokenized text, and moved tensors across process boundaries. One batch had 15-20 seconds of training after more than 100 seconds of fetching. The speaker's wait-time ratio, wait time divided by total pipeline time, was 85%.

The first fixes. The team added concurrency. Async I/O fetched images from S3 in parallel. Ray actors, a process-based worker pool, handled CPU-heavy decoding and preprocessing. Per-batch data time dropped to roughly 40 seconds, a 25% gain. Then a background thread prefilled a queue, keeping the producer ahead of the trainer. Fetch and transform time effectively hit zero per batch. Neither trick is exotic, but each must match the bottleneck.

The hidden cost. Once loading and transforms stopped blocking, transport emerged. Each sample carried megabytes of image arrays, and every sample was pickled on a worker, copied to the coordinator, repickled onto the trainer, every step. The data generator could outrun the trainer, but the memory copy bill was huge. Ray's object store fixed it: workers returned pointers instead of values, while Megatron ranks fetched directly. This eliminated a put-object call and cut overhead. Each parallelism choice adds communication cost.

Scale changed the answer. The same pipeline regressed in production after scaling to four times the data parallel ranks, 200 data streams, and a 4x batch size. Ray's default scheduling collocated workers, so one node's network interface card became the constraint. Tuning zero-copy retrieval and spreading workers across nodes did nothing at small scale, but together delivered a 50% throughput gain at scale. A config that looks neutral in a smoke test can dominate production. The team needed both flags, not one.

What to trust. The evidence is one production case study with concrete numbers but no full environment details. There are no hardware specs, cluster size, cost figures, or reproducible benchmark. The 90% utilization claim is plausible but not independently verified. The patterns themselves are well-established: prefetching, concurrency, object stores, spread scheduling. The novelty is the packaging for multimodal training, not the underlying techniques. Read this as a useful case study, not a benchmark.

The takeaway. The biggest wins came from profiling the baseline and asking whether the system was CPU, I/O, compute, or memory bound. Solving one bottleneck exposed the next: throughput, then wait time, then memory, then retrieval cost. For teams doing custom multimodal training, this is a practical checklist. For everyone else, the signal is that GPU utilization is an infrastructure and workflow problem, and the data path can pin it.