Voice agents in production: cascading pipelines beat speech-to-speech

Voice agents are transitioning from demos to production, and two builders agree: cascading pipelines (ASR-LLM-TTS) remain the default for production. The flexibility to swap models per layer and route around provider failures outweighs the latency win of speech-to-speech. But the real bottleneck is context management and the unbearable cost of API calls.

The one-to-one-and-a-half-second latency budget is the hard constraint; everything else—model choice, reasoning depth, turn-taking complexity—gets traded against it. One panelist reports 500-700ms P50 transcription times, leaving room for LLM and TTS without exceeding 1.5s. Yet most builders still guess at where costs go.

Fallbacks per layer, dynamic routing, and KV cache trade-offs separate production-grade from demo-grade. Open-source LLMs are now 'as good as' Claude for some builders, but GPU costs remain high. TTS evaluation remains entirely vibe-driven—there is no quantitative standard for naturalness.

The biggest strategic gap: measuring task completion. Teams optimize latency and word error rate but skip the ROI question. Vertical solutions sidestep this by narrowing goals; horizontal platforms still rely on human vibe checks.