Voice AI: The hard parts are pipeline design and latency, not models

The state-of-the-art for voice agents remains a cascaded pipeline: speech-to-text, LLM, then text-to-speech. Full speech-to-speech models exist but lack the reliability, interpretability, and guardrails needed for enterprise deployments. The tension is clear: naturalness vs. control. Teams trade latency for intelligence, and reliability for speed. Hype around end-to-end voice models ignores the real operational challenges.

Latency is the hardest problem. Teams shave milliseconds across every pipeline step, using fillers and parallelization to maintain conversational flow. Cost compounds when callers hang up early. Workflow design—not model choice—determines success. Inbound vs. outbound use cases demand radically different architectures, from rigid guardrails for debt collection to flexible retrieval for customer support.

Context compaction is a solved problem in voice, yet coding agents are only now discovering it. Turn-taking is surprisingly hard. Multilingual support requires component swapping, not model retraining. The real innovation has been in cascaded orchestration: using small models for classification, thresholds for context, and fillers for latency masking.

The core insight: most teams want a single model to solve everything. The reality is that product engineers and FDEs already know the best systems are hybrid. Speech-to-speech for the active loop, cascaded for complex tool calls. The next frontier isn't a better model—it's better orchestration and evaluation.