Voice is the least forgiving interface an agent can have. In text, thinking time is invisible; on a phone call every millisecond is audible, and a pause carries meaning the system did not intend. Most of our voice research is therefore about time rather than about language.
The pipeline
Transport
- SIP / PSTN
- WebRTC
- jitter buffer
- echo cancellation
Turn detection
- VAD
- endpointing
- barge-in
- backchannel filtering
Transcription
- streaming STT
- partial hypotheses
- finalisation
Reasoning
- context assembly
- model
- tool calls
- guardrails
Synthesis
- streaming TTS
- clause chunking
- playback queue
Two of these are not compute. Transport is a network path - for a PSTN call it includes a carrier leg and a jitter buffer at each hop - and turn detection is a *prediction*, not a measurement. Both are routinely larger than the model call and both are missing from most latency discussions.
Turn detection and barge-in
Voice activity detection asks whether someone is speaking. Endpointing asks whether they are finished - a question about the future, since a thinking pause and a completed sentence are identical in the audio.
Interruption handling needs three outcomes, not two. Between "keep speaking" and "stop" sits ducking: lower the volume, hold the audio queue, and wait to see whether the caller actually took the floor. Most backchannels - "mhm", "yes", "genau" - resolve inside that window, and treating them as interruptions is what makes an agent feel skittish.
When the caller cuts in, the conversation history must be truncated to what was actually played, not to what was generated. Otherwise every later turn reasons against sentences the caller never heard. We built this in the audio layer first and every bug we chased came from that decision.
Latency
The caller measures exactly one interval: from the end of their sentence to the first syllable of the reply. Everything after that first syllable is invisible - which is the most exploitable fact in voice engineering, and the reason clause-level streaming into TTS is the single highest-leverage change in most pipelines.
We maintain an explicit stage-by-stage budget rather than a single target, because a single number cannot tell you which stage to fix. The latency budget experiment sets it out in full, and the pipeline it describes is the one running behind bitpull.ai, where the constraints stop being theoretical.
Tool calls inside a live conversation
A calendar or CRM lookup takes long enough to be heard, and silence on a phone line reads as a dropped connection. The fix is conversational rather than technical: narrate before awaiting, with a line derived from the tool being called so the agent cannot describe an action it is not taking.
Transport: SIP and WebRTC
| SIP / PSTN | WebRTC | |
|---|---|---|
| Audio bandwidth | Narrowband typically | Wideband |
| Path length | Carrier hops, gateway | Direct or via one relay |
| Echo conditions | Speakerphones, handsets, varied | Browser AEC, usually good |
| Identity signal | Caller ID - spoofable | Session, optionally authenticated |
| Failure mode | Degrades audibly | Drops or reconnects |
Open questions
- Does a speech-to-speech model beat a cascade once you account for what the cascade's text layer gives you - inspectable guardrails, logging, tool-call interception?
- Can semantic endpointing - deciding whether the utterance is a complete thought - pay for its own added latency?
- How do you evaluate "naturalness" without a listening study, given that transcription accuracy plainly does not capture it?