Voice-In, Visuals-Out: The Latency Hack That Makes Agents Work

Allen Pike argues that while voice may be the natural input for AI, voice output imposes brutal latency requirements—200ms for seamless conversation. That's nearly impossible with today's inference chains. His team at Forestwalk therefore switched to a voice-in, visuals-out pattern: spoken commands trigger visual responses within one second, staying inside human attention budgets.

This shift unlocks real-time meeting agents that can take action, like filing a Linear issue, as people speak. The approach works because visual latency tolerance is far more forgiving than speech-to-speech exchanges. Pike calls it 'the agony and the ecstasy' of latency.

Making this feel delightful demands engineering discipline. You need a small, fast model (Haiku-class) on a low-latency platform, eager inference every 1–2 seconds without waiting for silence, and aggressive prefix caching so the first 90% of the context stays identical across requests. Without these, the experience breaks down.

The takeaway: the path to practical voice agents doesn't require novel architectures; it requires choosing the right output modality and tuning infrastructure around strict latency budgets. This pragmatic pattern could change how teams build meeting copilots and real-time tooling.