12 Trillion Synthetic Tokens: The Infrastructure Lessons DatologyAI Learned

The scale achieved. DatologyAI, co-founded by Bogdan Gaza, completed a synthetic data run of 12 trillion tokens across web, math, and code. Their approach, Beyond Web, rephrases high-quality data points from customer datasets rather than generating from scratch. Results show matching larger models with smaller ones and reducing token requirements by 2.7x–5.3x, but the talk focuses on the engineering behind the pipeline.

The metadata bottleneck. Before any GPU work, the team needed to list all input files from S3. With millions of partitions, sequential listing would take 9–11 days—unacceptable on expensive GPU clusters. By switching to batched S3 list API calls (up to 1,000 per request), they cut this to two hours. The lesson: prefetch and batch metadata before scheduling any job at trillion-token scale.

GPU instability and recovery. Long-running synthetic data jobs inevitably hit bad GPUs or node failures. Without checkpointing, an eight-hour partition failure in the last five minutes wasted all work. DatologyAI right-sized partition durations and added periodic checkpointing to S3. Failed jobs now retry with minimal lost progress, turning multi-day failure scenarios into manageable retries.

Cross-cluster orchestration. The team runs curation (Spark/Ray) on CPU clusters and generation on separate H100 GPU clusters, often across multiple providers. CPU and GPU needs for the same job must be scheduled atomically, or bin-packing problems cause idle resources. They built an internal scheduler that reserves CPU pools (for Ray heads) and GPU pools together, decoupling but coordinating allocation across Kubernetes clusters.

Inference optimization. Batch inference for synthetic data differs from online serving. By grid-searching VLM and SGlang parameters—batch size, speculative decoding, and other flags—they gained 40% throughput. This required a disciplined benchmarking harness, not ad-hoc tuning.

Implications for teams. Synthetic data at this scale shifts the bottleneck from data quality to operational reliability. Metadata management, checkpointing, cross-cluster scheduling, and inference tuning become first-class concerns. Teams planning similar pipelines should budget infrastructure engineering alongside data science.