Build a Voice Agent in an Hour with Claude Code | AssemblyAI Workshop

What changed: AssemblyAI now offers a vertically integrated Voice Agent API (STT+LLM+TTS) over WebSockets, designed to plug directly into your app and UI. In the workshop, Claude Code scaffolds a Python backend (token issuance) and a browser client that streams audio, shows transcripts, and handles tool calls—demonstrating that infra setup is no longer the main blocker.

Why it matters: Teams can bypass LiveKit/Pipecat orchestration and vendor-hosted “black box” agents, building bespoke experiences with their own UI and business tools (Calendly, databases, CRMs). The API’s simplicity shifts effort to conversation design, tool orchestration, latency policy, and monitoring—where outcomes are actually determined.

Tradeoffs: This is POC-friendly, not production-ready out of the box. Data retention is currently off (session history is “coming soon”); telephony/SIP is early-access; TTS is only available via the voice agent, not standalone; non-English/dialect voices are limited. Pricing claims were vague, and real enterprise needs (BAA, compliance, audit) require direct contact.

Operational consequences: Tool calling remains the brittle point. The recommended “progressive tool reveal” pattern (check first, then enable booking) and minimizing tools improves accuracy and reduces hallucinations. Latency tuning is a governance decision: lowering min/max silence reduces wait time but increases interruption risk; enable AEC if not using headphones. Use real call transcripts to ground prompts.

What teams should watch: Instrumentation, post-call scoring, and QA are not built-in; you’ll likely use their LLM Gateway or your own analytics for summaries and scoring. Plan for concurrency limits, cost controls, failure fallbacks, PII handling, and telephony integration timelines. Overreliance on agent-generated scaffolds can mask hidden complexity—own the code and tests.