Voice Agents That Don't Break: Measure Conversations, Not Models

The core problem. Most voice agents fall apart when interrupted because teams optimize for model-level benchmarks rather than conversation-level performance. Jesse Hall argues that leaderboard scores for speech-to-text, LLM, or text-to-speech models don't predict real-world behavior when a caller speaks over the agent or changes their mind mid-sentence. The errors cascade: a misheard date or amount flows downstream, and the agent never recovers.

The missing metric. Teams commonly measure time-to-first-byte or time-to-first-token, but these don't correspond to what the caller hears. Hall cites cases where a TTS provider reports 210ms time-to-first-byte but then delivers 750ms of silence before actual audio. The real metric is time-to-first-audio. Similarly, LLM speed should be measured as time-to-first-sentence, not token generation rate, because TTS needs a full sentence to begin speaking.

The latency budget. There is a fixed window between the caller's last syllable and the agent's first audible response. Hall puts the acceptable bound at roughly 1.5 seconds; beyond that, the conversation feels off. Below 500-600ms, it becomes human-like. Every model choice, function call, and handoff must fit within that budget. The goal isn't raw speed—it's maximizing human-likeness within a fixed latency constraint.

Async tool calls. A common failure is pausing the conversation while a backend tool runs (e.g., looking up an account). Hall demonstrates a simple code change: detaching tool calls from the main thread so the agent can continue speaking while waiting. In a demo of a hotel booking agent, the system saves data, queries availability, and moves to the next question without awkward silence. If a tool errors, the agent corrects itself later.

Turn detection beyond transcripts. Most turn detectors rely on transcript pauses, causing premature interruptions (e.g., cutting off after "500" when the user meant "500 and 55 cents"). LiveKit's detector also analyzes audio rhythm, pitch, and intonation. Hall claims their approach reduces unwanted interruptions from 9.9% to 4.5% in their tests. This kind of nuance is absent from standard benchmarks but critical for natural conversation.

Testing the whole conversation. The only reliable way to choose models is to run a complete agent against simulated callers that interrupt, mumble, or add background noise. Hall's open-source methodology runs identical scenarios with different model stacks and measures business outcomes (e.g., correct reservation saved, correct amount charged). The model that wins this test may not top any leaderboard—as shown by Gemma 4 31B performing best on LiveKit's inference but slower on another provider.