A realtime voice agent is a pipeline that runs while its input is still arriving. That single property - nothing waits for completion - is what separates it from a system that transcribes a recording and answers afterwards.
Transport
- SIP / PSTN
- WebRTC
- jitter buffer
- echo cancellation
Turn detection
- voice activity detection
- endpointing
- barge-in
Transcription
- streaming STT
- partial hypotheses
- finalisation
Reasoning
- context assembly
- model
- tool calls
Synthesis
- streaming TTS
- clause chunking
- playback queue
Transport is not a detail
A phone call arrives over SIP from a carrier network, usually with a narrowband codec and a jitter buffer at each hop. A browser call arrives over WebRTC, wideband, with a shorter path. The same agent sounds different on the two, and callers attribute that difference to the AI.
Turn detection is the hard part
Voice activity detection asks whether there is speech. Endpointing asks the much harder question of whether the speaker is *finished*. A pause for thought and the end of a sentence look identical in the audio energy; distinguishing them requires either a timer, which is a compromise, or the words themselves, which costs time.
Everything overlaps
Transcription runs while audio arrives. Context assembly can start on a stable partial transcript. Synthesis starts on the first clause the model emits, not the last. Done properly, the caller starts hearing an answer while the system is still generating it.
This is also why voice systems are hard to debug: at any moment four stages are in flight, and a fault in one surfaces as a symptom in another.
The alternative architecture
Speech-to-speech models take audio in and emit audio out, collapsing transcription, reasoning and synthesis. They remove stages and latency, and they remove the text in the middle - which is where guardrails, logging and tool-call inspection currently live. Which trade wins is genuinely unsettled; we run the cascade in production and watch the alternative closely.