In Voice AI, Latency Is the Whole Experience
A pause that reads as thoughtful in a chat window reads as a dropped call on the phone. The gap between those two reactions is the entire engineering problem.
The short version
4 things that decide this
- 01A voice agent's reply passes through three stages in sequence: speech-to-text, model inference, and text-to-speech. Each one adds delay before the caller hears anything.
- 02Human turn-taking runs on a gap of roughly 200 milliseconds on average, found across ten languages by Stivers et al. (2009). We treat that as the budget: a voice pipeline much slower than it gets talked over.
- 03Streaming is what makes a slow pipeline sound fast: each stage starts working on partial output from the stage before it, instead of waiting for it to finish.
- 04A demo run on a quiet line with a short scripted question hides queueing, retries and a model that runs slower on a longer prompt. Ask what the pipeline does after word one, not before it.
Chat and voice are not the same latency budget
In a chat interface, a three-second wait reads as the assistant thinking. There is a typing indicator, and the reader has other things on the screen to glance at. Nobody hangs up on a chat window.
A phone call has no such slack. Stivers et al. (2009) studied ten languages and found the gap between conversational turns runs near 200 milliseconds on average. Most transitions land within a quarter-second of that mean. A longer gap reads as confusion, distraction or a dropped line. A voice agent that takes two seconds to answer is not being thoughtful. It is being slow, and the caller reacts as they would to a person who stopped paying attention.
This is why a voice product cannot borrow its latency budget from a chat product, even when both run on the same underlying model. The interface changes what the same delay means to the person on the other end.
Three stages, stacked in sequence
A voice agent's answer is not one system. It is three, run one after another. Speech-to-text turns the caller's audio into words. The language model reads those words and decides what to say. Text-to-speech turns that decision back into audio the caller can hear. Every stage adds its own delay, and those delays add up rather than overlap unless the pipeline is built to overlap them.
Transcription is usually the fastest of the three. Even in real time, though, it has to wait for the caller to stop talking before it can be sure a sentence is finished. Ending that wait too early cuts the caller off. Waiting too long adds a pause the caller can feel.
Inference is where most of the variance lives. A short scripted question and a longer, messier one take very different amounts of time from the same model. Add a tool call, a database lookup or a knowledge base search in the middle of answering, and the gap grows. A demo built around a short, predictable question never shows this.
Synthesis then has to turn text into audio that sounds like a person, not a robot reading a script. Higher-quality voices generally cost more time to generate, which is a real trade-off, not a solved problem.
- 01Speech-to-text: fast per word, but it has to decide when a sentence is actually finished.
- 02Inference: the least predictable stage once a tool call or a lookup runs mid-answer.
- 03Text-to-speech: higher voice quality generally costs more generation time.
- Caller speaksAudio streams in as it is captured, not after the sentence ends.
- Streaming transcriptionPartial text reaches the model before the caller finishes talking.
- Model starts on partial inputEarly tokens generate while later words are still arriving.
- Streaming synthesisAudio for the first words plays while the model writes the rest.
- Caller hears a replyThe full round trip can still take over a second. It just does not sound like it.
Each stage overlaps the one before it instead of waiting for it to finish. That overlap is most of what separates a voice agent that feels responsive from one that does not.
What to ask before you trust a demo
A demo call answers one question well: can this thing hold a conversation. It does not answer the question that decides whether the system survives production, which is what happens to the pause when the question gets harder.
Ask what happens when the answer needs a lookup: a customer record, an order status, an available time slot. That round trip sits inside the inference stage and can add real time. The honest answer names how the pipeline covers it, not whether it exists.
Ask whether transcription, inference and synthesis are streamed or run one after the other. A pipeline that waits for each stage to finish before starting the next one is adding its delays together. One built to overlap them is hiding the same total work behind a faster-feeling reply.
Ask what the system does with silence. A caller who pauses to think should not be cut off. Someone who has finished talking should not sit through a needless delay while the system decides. Both directions of that judgement call show up as latency the caller feels.
Questions this raises
01What is an acceptable latency for a voice AI agent?
Human conversation runs on a turn-taking gap near 200 milliseconds on average, found across ten languages by Stivers et al. (2009). On ZhoopZhoop's phone lines we budget the full pipeline, transcription through synthesis, well under a second. Callers start talking over anything slower, or ask if the call dropped. What matters is the full round trip from the caller finishing a sentence to audio starting, not any single stage measured alone.
02Why does my voice agent pause before answering?
The pause is almost always the inference stage, especially when the answer requires a tool call or a database lookup instead of a fixed response. Streaming hides most of that wait. Partial text reaches speech synthesis as soon as the first words are ready, instead of after the full answer is done.
03How do you design a real time voice pipeline?
Stream every stage instead of running speech-to-text, inference and text-to-speech one after another. Partial transcription reaches the model before the caller finishes speaking. The model starts generating before it has the full request. Synthesis begins on the first words while the rest of the reply is still being written. The stages overlap instead of stacking.
