Small models are ready. Your serving stack isn't.
Small models now match frontier quality on specific workloads.
The bottleneck is the serving stack: routing inefficiency, silent bugs, and organizational friction waste the gains.
The real bottleneck. Inference infrastructure is now the limiting factor for production AI.
Small models suffer routing bottlenecks that drop GPU utilization to 20%.
Mamba inference engines hide silent bugs that appear only under memory pressure.
Agent workloads expose caching inefficiencies that vendors claim to address.
Teams should shift budget from model selection to infrastructure engineering.
Small model surprise. Open-weight models under 30B parameters match frontier quality on classification and routing tasks.
Jeff returns structured probabilities at 200x speed and 400x cost reduction.
Qwen 3 approaches GPT-4.1 on dedicated workloads.
But each model demands workload decomposition and infrastructure tuning for automatic gains.
Hidden costs. Silent inference bugs and AI-infra team friction are the hidden costs of multi-model adoption.
AI21 engineers found a vLLM bug producing gibberish after thousands of requests without warning.
AI engineers want to deploy fine-tunes fast; infra engineers need to tune the stack per model.
Teams need log-probability tools and scheduling abstractions before scaling.
My take
Inference infrastructure is where the hard work starts now.
The small-model cost gains are real, but they demand infrastructure engineering that a lot of teams haven't invested in.
The log-probability comparison script from the vLLM bug hunt is a cheap win that protects against silent regressions.
Start there.
The pull-based scheduling fix is for later, when you outgrow a single model.
And Jeff's waitlisted API warrants a pilot, not a platform bet.
The videos behind this brief
We watched these so you don't have to.
Video and thumbnails remain the property of their respective creators. tldw.news provides editorial analysis, commentary, and discovery links to original content.