Deepgram
Fast enough for a live call, if you tune the pauses
We run Deepgram's Voice Agent API in ZhoopZhoop, a phone-answering and parts-procurement system for a multi-branch auto repair business. This is what real caller audio taught us.
The short answer
Deepgram's Voice Agent API is the right default for a phone agent that has to listen, decide, and reply while the caller is still talking, provided you tune endpointing against your own callers instead of shipping the default.
Choose differently if your job is transcribing recordings that already exist rather than answering live. A batch transcription model reads the whole file at once and does not have to guess, mid-sentence, whether someone has finished talking. That guess is the hard part of a phone agent, and it is the part Deepgram is built around.
What decides this
4 things that decide this
- 01Deepgram bundles speech-to-text, text-to-speech, and turn-taking into one Voice Agent API, so the pipeline that used to be three vendors is one WebSocket connection.
- 02Endpointing, the setting that decides when the agent thinks you have stopped talking, is the single biggest lever on how the call feels. The default is tuned for a general caller, not yours.
- 03A caller who trails off, coughs mid-sentence, or reads a part number aloud digit by digit breaks the default timing before anything else does.
- 04It sits inside a chain: Twilio carries the audio, Deepgram turns it into text and back into speech, and a separate model handles the reasoning. Latency is a property of all three, not one.
Speech in, an agent's turn to talk, speech out, on one connection
Deepgram's Voice Agent API streams a caller's audio in and transcribes it as it arrives. It decides when their turn has ended, then streams synthesized speech back out. Older phone-agent stacks wired this together from three separate products: a speech-to-text vendor, a text-to-speech vendor, and hand-rolled logic for when to interrupt or wait.
The part that actually differs from a transcription API is turn-taking, also called endpointing. It has to decide, while someone is mid-sentence, whether they have finished a thought or just paused for breath. Get that wrong and the agent talks over the caller, or leaves a silence long enough that the caller starts repeating themselves.
Where it holds up and where it does not
Strengths
- One connection replaces three vendors. Speech-to-text, text-to-speech, and turn-taking arrive already wired together, which is real integration work removed from the build.
- Streaming transcription is fast enough that a caller does not notice the gap between finishing a sentence and hearing a reply, on the calls we have logged.
- It plugs cleanly into Twilio's media streams, so the audio transport and the speech layer do not fight each other.
- Function calling is built into the agent loop, so the model can trigger a lookup or a booking action mid-conversation rather than waiting for the whole reply to render first.
Trade-offs
- Default endpointing is tuned for a generic caller. A shop customer reading a license plate or a part number aloud, one character at a time, gets cut off before the default settings, and we had to raise the endpointing timeout for that flow specifically.
- Background noise on a real shop floor degrades transcription accuracy. Engines running and tools going never show up in clean test audio, so we caught this only after routing live calls.
- Latency is a chain, not a single number. Deepgram's own leg is fast, but Twilio's transport and the reasoning model's response time both sit in the same round trip, so a slow function call anywhere downstream still reads to the caller as Deepgram being slow.
- Turn-taking tuning is closer to product design than configuration. There is no single setting that is correct for every caller population, and getting it right took listening to failed calls, not reading documentation.
- Caller pausesReading a number, catching a breath
- Endpointing guessesDefault timeout treats the pause as done
- Agent replies earlyTalks over the rest of the sentence
- Caller repeatsCall runs long, trust drops
- Timeout tuned per flowFixed once we logged the failure
The failure is never the transcription. It is the guess about when someone has finished talking, and it only shows up once real callers, not test scripts, are on the line.
What ZhoopZhoop taught us about running this live
ZhoopZhoop is a multi-branch auto repair business whose phone used to go unanswered whenever every technician was under a car. Its inbound agent runs on Deepgram for speech and turn-taking, Twilio for call routing, and OpenAI for reasoning and function calls. The same stack drives outbound calls too. The agent rings suppliers, asks for parts availability and pricing, and hands back a comparison. A job that used to mean dialing shop after shop now runs unattended.
The two problems worth naming both trace back to endpointing. Callers reading a part number or plate digit by digit paused longer between characters than the default timeout allowed, so the agent jumped in mid-number. Outbound calls to suppliers needed a longer silence tolerance too, since a person checking a shelf often puts the phone down mid-call. The fix was different endpointing settings per call type, not one global default. Expect the same if you build on this.
- 01Log every call transcript against the audio. Endpointing failures are audible in a recording and invisible in a transcript alone.
- 02Tune endpointing per call flow, not once for the whole agent. Inbound customer calls and outbound supplier calls behave differently.
- 03Treat background noise as a design input, not an edge case. A shop floor is loud, and test audio recorded in a quiet room will not catch it.
The system running on this stack
What teams ask before committing
01Deepgram or Whisper for a voice agent?
Deepgram, if the agent has to reply while the caller is still talking. Whisper transcribes a finished recording well and cheaply, but it was not built to stream a decision back in real time. A voice agent that answers the phone needs the streaming and turn-taking Whisper does not provide out of the box.
02Is Deepgram's Voice Agent API ready for production phone calls?
Yes, with the caveat that endpointing needs tuning against your own callers before launch. We run it in one live phone-answering system without an outage traced to Deepgram itself. Every failure we hit was a default setting that assumed a caller who does not exist in an auto repair shop.
03What causes a voice agent to talk over the caller?
Endpointing set too aggressively. The agent decides a pause means the caller is finished and starts replying before the sentence actually ends. It shows up most on callers reading numbers aloud, where the natural pause between digits is longer than ordinary speech.
04Does background noise break Deepgram's accuracy?
It degrades it. Transcription accuracy on our shop-floor calls is measurably worse than on the clean audio most demos use, because tools and engines run behind the caller. Test with real environmental noise before you trust a latency or accuracy number from a quiet room.
05What else does a Deepgram-based phone agent need?
A call-routing layer and a reasoning model, at minimum. Deepgram handles speech in and speech out, but it does not carry the phone call or decide what the agent should say next. We pair it with Twilio for routing and OpenAI for the decision logic. Measure latency across all three as one system, not Deepgram alone.
Related
- AI agent development →Where we build systems like this one.
- What an AI agent is →The distinction between an agent and a scripted workflow.
- Should an AI receptionist handle emergency calls? →Where the handoff to a human has to happen.
- Automotive software development →The industry ZhoopZhoop's system runs in.
