Voice AI without the cringe: latency, prosody, and tools
Most voice AI sounds bad. The 2019 IVR ("press 1 for sales") is bad. The 2023 chatbot-with-TTS is also bad — same robot, different voice. The 2025 generation finally crosses some thresholds.
Here's what we found mattered, in order:
**Latency under 700ms p95.** Humans expect a turn to come back inside about 600ms in casual conversation. Above 1 second, they assume the line dropped or the agent didn't understand. We measured ours obsessively: 540ms median, 680ms p95. That number is the difference between "ok this is a person" and "wait am I talking to a bot."
How we got there:
- Vapi handles the streaming voice loop with sub-100ms STT chunks
- We pipe partial transcripts to Claude with a custom "incremental decode" prompt that lets us start reasoning before the user is done speaking
- Tool calls execute in parallel with response generation when the next response is predictable
- TTS is streamed from the first phoneme, not the first sentence
**Prosody — the rhythm and intonation.** Even with sub-second latency, monotone TTS sounds like a robot. We use Vapi's ElevenLabs integration with a "warm receptionist" voice cloned from a real human who agreed to license it. The voice has natural pauses, breath sounds, mid-sentence emphasis. Customers don't consciously notice; they just don't hang up.
**Tools — what the agent can actually do.** This is the unsexy part. Herald has 8 tools:
- look_up_inventory(make, model, year_range, max_price)
- check_test_drive_availability(rooftop_id, time_window)
- book_test_drive(customer, vehicle, time)
- schedule_service_appointment(customer, vehicle, service_type, time)
- transfer_to_human(reason, urgency)
- send_sms(to, body)
- log_lead(name, phone, intent)
- get_dealer_hours(rooftop_id)
Every tool has typed inputs, typed outputs, idempotency keys, and error fallbacks that the agent knows how to recover from in conversation.
The naive build: connect a voice loop to a chatbot prompt and hope for the best. The result of that build is what most dealers tried in 2023 and abandoned. The agent runs into something it can't handle and hangs up. Or worse, it pretends.
The real build: voice as a thin shell over an action-oriented agent with proper tool design. The voice model is the smallest part of the system. The tools are 90% of the work.
What we still hate: handoff to a human. We have the mechanical pieces working — the agent transfers with full context, the human sees the transcript, the line stays warm. What we don't have yet is "the agent recognizes a vibe shift early and proactively warms a human." That's the next 90 days.