The first instinct when a voice agent feels slow is to look at the model. It is the most visible component and the one with a number attached to it. It is also, in a well-built pipeline, rarely the largest contributor to the gap the caller actually perceives.
This piece describes how we think about that gap: what it is made of, which parts can be overlapped, and which parts are irreducible because they encode a decision rather than a computation.
What the caller measures
The caller does not measure your pipeline. They measure one interval: from the moment they stop speaking to the moment they hear the first syllable of the reply. Everything that happens after that first syllable is invisible to them, which is the single most exploitable fact in voice engineering.
- 01Caller stops
- 02Endpointing decides
- 03First model token
- 04First audio chunk
- 05Caller hears
Note what is *not* in that chain: generating the rest of the answer, synthesising the rest of the audio, writing the transcript, updating the CRM. All of that can happen while the caller is already listening.
Rule 1 - stream at clause boundaries
The highest-leverage change in most pipelines: do not wait for the model to finish before synthesising speech. Cut the token stream at the first clause boundary, send that clause to TTS, and start playing audio while the model is still generating.
The effect is that first-audio latency stops depending on answer length. A three-sentence answer and a one-sentence answer start speaking at the same moment. Without this, every improvement elsewhere is fighting a variable it cannot control.
Once audio is queued, an interruption has to drop that queue and truncate the conversation state to what was actually played. If you skip the second half, the agent continues reasoning against sentences the caller never heard. See barge-in without false interruptions.
Rule 2 - overlap the preparation
Context assembly, retrieval and state loading do not need a final transcript. They need a *good enough* one. Once the incremental transcript has stabilised - successive hypotheses agreeing on all but the last word - start the retrieval. If the final transcript differs materially, discard and redo. In practice it rarely does.
The more aggressive version is starting generation on the partial transcript and cancelling if endpointing resolves differently. This is worth doing only if the cancellation path is genuinely complete. A half-wired cancellation produces two overlapping replies, which is worse than any latency you were trying to save.
Rule 3 - endpointing is a policy, not a bug
Deciding that the caller has finished speaking is a prediction about the future. Every millisecond you shave off the silence threshold buys latency and costs interruptions. There is no setting that is correct in general.
| Deployment | Caller behaviour | Threshold |
|---|---|---|
| Reception / booking | Short, complete utterances | Aggressive - latency wins |
| Technical support | Pauses to read serial numbers | Conservative - interruption is costly |
| Outbound survey | Considered answers, thinking pauses | Conservative |
| Menu replacement | One or two words | Very aggressive |
Semantic endpointing - asking whether the partial transcript is a *complete thought*, not just whether the audio went quiet - is the direction that makes this less of a trade-off. It is also more expensive per turn, which puts it back in the latency budget it was meant to relieve.
Rule 4 - never leave silence during a tool call
A calendar lookup or a CRM query takes long enough to be audible. Silence on a phone line reads as a dropped connection, and callers hang up on it. Speak first, then await: emit a short line derived from the tool being called, and only then block on the result.
Derive the filler from the tool, never from the model. A generated filler will eventually claim the agent is doing something it is not, and on a phone call there is no way to take that back.
What we would do differently
We built interruption handling as an audio-layer concern and spent weeks on bugs that were all the same bug: the audio layer and the conversation state disagreeing about what had been said. Interruption is a state problem that happens to involve audio. Build it there.
We also over-invested in shaving the model call before instrumenting the stages separately. Once each stage reported its own timing, the actual distribution was not where we had assumed it was - which is the ordinary outcome of measuring after optimising.
The open question
Everything above describes a cascaded pipeline: speech in, text through a model, speech out. A speech-to-speech model collapses several of these stages and invalidates parts of this analysis. Whether the cascade's controllability - you can inspect and constrain the text in the middle - is worth its latency is not settled, and we do not think it is settled by anyone else either.