Speech-to-Speech vs Cascaded Voice Pipeline
One model that turns audio into audio, against three models chained together. The naturalness gap is closing, but the debugging gap is not.
The verdict
Choose a cascaded pipeline (speech-to-text, then an LLM, then text-to-speech) for a production voice agent you need to test and debug. Pick speech-to-speech, such as OpenAI's Realtime API, when raw naturalness and interruption handling matter more than seeing what the model heard.
We build phone agents with a cascaded pipeline. Twilio takes the call, Deepgram handles speech-to-text, an LLM reasons, and a separate step handles text-to-speech. That choice is not caution about newer models. It lets us run evals against a real transcript before launch. It also lets us swap one component without touching the rest.
The newer models have closed most of the latency gap that used to make cascaded the only sane choice. What they have not closed is the transcript. A single model that turns audio directly into audio gives you no clean text checkpoint to log, score, or hand to an evaluator. That trade is fine for a demo. On calls that matter, it decides whether you can find out why the agent said what it said.
Where they genuinely differ
Compared on what changes a production build, not on demo naturalness.
| Dimension | Speech-to-speech | Cascaded |
|---|---|---|
| Latency | Lower. Audio goes in and comes out of one model, often under half a second to first response. | Higher. Each step adds its own delay; a well-tuned pipeline still runs one to three seconds end to end. |
| Interruption handling | Native. The model hears overlapping speech and reacts inside the same audio stream. | Bolted on. You build turn-taking logic on top of speech-to-text endpointing signals. |
| Debugging a bad call | Hard. There is no separate transcript step to inspect; you are reading model behaviour, not text. | Direct. Every stage produces a log: what was heard, what the LLM decided, what was said back. |
| Running evals before launch | Limited. Few eval frameworks score raw audio-to-audio behaviour well yet. | Established. Text transcripts plug into the same eval tooling used for any LLM agent. |
| Swapping vendors | Locked to one provider's voice model and its update schedule. | Open. Change the speech-to-text, the LLM, or the voice independently. |
| Best fit | Consumer-facing voice products where conversational feel is the product. | Business phone agents that book appointments, answer questions, or route calls. |
Speech-to-speech
Where it wins
- Handles interruptions and overlapping speech the way a human conversation actually works.
- Carries tone, pauses, and emotion through the model instead of flattening speech into text first.
- Fewer moving parts to wire together, since one API call replaces three separate services.
Where it hurts
- No clean transcript checkpoint, which makes systematic evals before launch difficult to run.
- Debugging a wrong answer means inspecting model behaviour, not reading a log line.
- Ties the whole voice experience to one vendor's model, pricing, and release schedule.
Cascaded pipeline
Where it wins
- Every stage produces a log, so a bad call can be traced to the exact step that failed.
- Text transcripts plug into standard eval tooling, so you can score an agent before it ever takes a real call.
- Each component, speech-to-text, the LLM, the voice, can be replaced without rebuilding the others.
Where it hurts
- Adds latency at every handoff between components, which a caller can notice as a pause.
- Turn-taking and interruption handling have to be engineered on top, not assumed.
- More services to run, monitor, and keep talking to each other correctly.
Rules that settle it
Work through these in order. The first one that matches is your answer.
- 01Choose cascaded if the agent has to pass an eval suite, get logged, or get audited before or after launch. That needs a transcript.
- 02Choose cascaded if the call can trigger a real action: booking a slot, updating a record, quoting a price. You need to see exactly what the model decided and why.
- 03Choose speech-to-speech if the product is the conversation itself, and a missed edge case costs you nothing more than an odd reply.
- 04It also fits a fast prototype, where you do not yet need the operational discipline of a business-critical system.
- 05Choose neither if the call only needs a fixed set of menu options. A scripted phone tree handles that without a model in the loop.
- Needs a transcript for evalsCascaded. Text is the checkpoint.
- Triggers a real action on a callCascaded. You need to see the decision.
- Naturalness is the whole productSpeech-to-speech wins on feel.
- Fast prototype, low stakesSpeech-to-speech, fewer parts to build.
- Fixed menu of optionsNeither. A scripted IVR is enough.
Both turn a phone call into a working conversation. The difference is whether anything you can inspect sits between the ear and the mouth.
Questions people ask next
01Is speech-to-speech ready for production phone agents?
It runs in production today for latency-sensitive, lower-stakes voice products. For agents that book, bill, or update real records, most teams still choose cascaded because it produces a transcript they can evaluate and audit. That balance is shifting as eval tooling for raw audio matures, but it has not shifted yet.
02Does cascaded really add noticeable delay?
A well-tuned cascaded pipeline, using a streaming speech-to-text model with fast endpointing, can keep total delay close to a second. The end-to-end models are still faster, often under half a second, and callers do notice the difference on a live line.
03Can I mix the two approaches?
Yes. Some teams run speech-to-speech for the open-ended part of a call. They hand off to a cascaded, tool-calling step the moment the conversation needs to trigger a booking or a lookup. The audio stays natural. The action stays auditable.
04What does 'observability' actually mean for a voice agent?
It means you can read what the system heard, what it decided, and what it said back, as text, after the call. A cascaded pipeline gives you that by default because speech-to-text produces a transcript. Speech-to-speech models process audio directly, so there is no equivalent text checkpoint to inspect.
05Will speech-to-speech eventually replace cascaded pipelines?
Audio-native evaluation tools would need to catch up to what text-based eval frameworks already do. Until a team can score a speech-to-speech agent as rigorously as a cascaded one, the two will keep serving different jobs. Neither is likely to replace the other soon.
