Small models are good. Your serving stack is the problem.

The premise. Small open-source models—those that fit on a single GPU—are no longer just toys. Daniel Svonava argues that models like Qwen 3 with 27 billion parameters now approach GPT-4.1 performance on specific tasks. The catch is that you cannot treat them as general-purpose drop-in replacements. Teams must decompose their workloads into slices and find the best open-source model for each slice: OCR in Vietnamese, question answering, image labeling, SQL generation.

The economic signal. The cost and latency advantages are dramatic. On an RTX Pro 6000, a single GPU can push half a million tokens per second through an embedding model with latencies in the tens of milliseconds. Compare that to managed embedding APIs that cost orders of magnitude more and take hundreds of milliseconds. For generative tasks like synthetic data generation or annotation, a small open-source model on your own hardware can deliver thousands of tokens per second per GPU, and scaling is linear.

The infrastructure problem. Serving a fleet of small models is fundamentally different from serving one large model. Traditional inference clusters use a top-down router that decides which worker gets each request. That works for large models with long-running requests. But with many small, fast requests, the router becomes the bottleneck. GPU utilization drops to 20-30% under constant load because the router cannot perfectly saturate workers. The core issue is that request batching happens at the wrong level.

The organizational friction. Adopting small models creates a communication bottleneck between AI engineers and infrastructure engineers. AI engineers want to serve custom fine-tunes and LoRAs they built last night. Infrastructure engineers need to tune the serving stack for each new model and hardware combination. The back-and-forth kills velocity. The tools that exist—vLLM, SGLang—are not pre-tuned for any specific model. Every adoption becomes a research project. Teams need an abstraction that lets AI engineers deploy models without infrastructure engineers tuning parameters each time.

The architectural solution. Superlinked's open-source cluster reverses the control flow. A lightweight gateway parses only enough of a request to attach metadata, then inserts it into a shared queue using NATS JetStream. Workers pull from that queue and form their own batches, which doubles throughput. The system supports multiple runtimes—PyTorch, Kendall, SGLang—wrapped behind a Rust sidecar. The key insight is that for small models, you want pull-based scheduling, not push-based routing. This lets workers self-saturate without a central bottleneck.