Voice Agents Fail on Names, Numbers and Noise
A demo call is a quiet room and a clear script. A real call has an impact wrench running behind the counter. Those are different engineering problems.
The short version
4 things that decide this
- 01Voice agents rarely fail on conversation. They fail on capturing a spelled name, a phone number or a part number correctly the first time.
- 02Background noise on a real call, a running compressor, an impact wrench, a shop radio, degrades speech recognition in ways a quiet demo never surfaces.
- 03The fix is confirmation design: the agent repeats back what it heard, in a format the caller can correct, before it writes anything down.
- 04ZhoopZhoop's voice agents answer calls for a multi-branch auto repair business, where names, plate numbers and part numbers arrive over shop-floor noise on every call.
The demo call is not the real call
Most voice agent demos happen in a quiet office, on a good phone line, with a caller who speaks clearly and expects to be understood. The agent handles it well, because the hard part of the job never shows up.
The hard part is not holding a conversation. Large language models are good at that now. The hard part is capturing one small piece of exact data correctly, on the first try. A customer spelling their last name over a bad cell signal. A phone number read out in a rush. A part number with a letter and three digits that sound like two other letters and three other digits.
Get the name wrong and the booking is under someone else. Get the number wrong and the callback goes nowhere. Neither failure looks dramatic in a transcript. Both cost a customer.
Noise is not an edge case in the shop
We built ZhoopZhoop's voice agents to answer inbound calls for a multi-branch auto repair business, and to call suppliers outbound for parts quotes. Both ends of that phone line are noisy in ways an office call is not.
A customer calling about their car is often standing near it. An impact wrench, a running engine, road noise if they're on the roadside. A supplier's counter has a radio on and other calls going. Speech recognition tuned on clean audio degrades against that kind of background. It degrades hardest on the words that matter most. Names carry no context to help a model guess them. Numbers sound similar to each other in a way normal words do not.
This is why we use Deepgram for speech-to-text and turn-taking, rather than treating transcription as a problem solved once and forgotten. Turn-taking matters as much as raw accuracy. A caller cut off mid-number will repeat it differently the second time, leaving the agent with two versions to reconcile.
- 01Names carry no context clues, so a mis-hearing has nothing to correct against.
- 02Numbers sound alike to a model in a way ordinary words do not.
- 03A caller cut off mid-sentence repeats it differently, which creates two conflicting records instead of one wrong one.
- Caller speaksName, number or part id, often over background noise.
- Speech-to-textDeepgram transcribes the audio in real time.
- Agent repeats it backSpelled out or read digit by digit, not summarized.
- Caller confirms or correctsThe agent waits for a yes before moving on.
- Record is writtenOnly after confirmation. Never before.
Every field that becomes a booking, a callback number or a parts order passes through this loop before it is saved.
Confirm before you write anything down
The fix is not a better transcription model. It is a rule. Nothing goes into a booking, a callback list or a parts order until the agent has said it back and the caller has confirmed it. A name gets spelled out letter by letter. A phone number gets read back in short groups a person can follow. A part number gets confirmed the way a pharmacist confirms a prescription: out loud, digit by digit.
This slows the call down by a few seconds. That trade is worth making. A booking under the wrong name is not a minor error. It is your customer, assuming the shop never called back.
ZhoopZhoop's outbound agents apply the same discipline in reverse. When they call a supplier for a parts quote, they read the requested part number back before ending the call. A mis-heard digit should never turn into the wrong part arriving days later. Staff can pull up any call's recording and transcript on the dashboard, across every branch, and check what was confirmed against what was booked.
Questions this raises
01Why does an AI phone agent mishear names so often?
A name has no surrounding context a language model can use to guess it, unlike a common word in a sentence. Background noise, accents and phone line quality all compound the problem. The fix is not perfect transcription, which does not exist, but a confirmation step that spells the name back before it gets recorded.
02How accurate is voice AI for phone number capture?
Accuracy depends heavily on call conditions, more than on the model itself. A quiet line produces a near-perfect capture. A noisy shop floor or a bad cell connection produces errors on individual digits. A callback number should always be read back and confirmed before the call ends.
03What should a voice agent do when it is not sure what it heard?
Ask, rather than guess. A well-designed agent repeats the detail back in a format the caller can correct, such as spelling a name letter by letter. It waits for a clear yes before moving on. Writing an unconfirmed guess into a booking or a callback list is the failure mode that costs a business a customer.
