Model Routing: The Orchestration Problem Your Team Will Face

Model routing promises 40% cost savings by using a frontier model as orchestrator and cheaper models for execution. The key insight: smarter models delegate better, making the overall system more cost-effective than simply routing to the cheapest model.

Naive task-based routing is fragile because task complexity changes mid-execution. The panel warns that without a frontier model maintaining oversight, small models can thrash, increasing cost and reducing quality. Context compaction and caching are critical to avoid token waste.

Co-designing models with orchestration in mind is the next frontier. Training models to collaborate, rather than treating them as standalone, could unlock further gains. But the field is young, and most solutions are still experimental.

Engineering leaders should watch for hidden costs: orchestration overhead, cache management, and the need for continuous tuning against real usage data. The promise is real, but the path requires system-level thinking, not just model-level optimization.