Realtime is a systems property, not a model property. A fast model in a badly arranged pipeline is slow, and a moderate model in a well-overlapped one is fast. The research question we keep returning to is not how quickly a model can answer but how much of the pipeline can be made to run at the same time.
How fast is fast enough?
Human conversational turn-taking runs on gaps of a couple of hundred milliseconds. That is not a target anyone chose - it is what people are used to, and departures from it carry meaning. A pause before answering reads as hesitation or disagreement, and callers apply that reading to a machine.
- Near the natural turn gap: reads as a conversation.
- Roughly half a second to a second: noticeably machine-like, still workable.
- Beyond about a second and a half: the caller starts talking again, assuming they were not heard.
Those boundaries are orders of magnitude from watching calls, not measured thresholds - the shape is robust, the numbers are soft. We watch them on live traffic rather than in a benchmark, because this research sits underneath an AI voice agent platform that answers real phone calls.
A steady 700 ms is easier to talk to than a system alternating between 200 ms and 1,400 ms. People calibrate to a rhythm; an unpredictable one prevents that and produces collisions. Optimise the tail before the mean.
Streaming as an architectural commitment
Streaming is usually adopted as a performance setting and then discovered to be an architectural decision. Once output has been delivered it cannot be revised: post-hoc validation no longer runs before the user sees the text, retries produce duplicates, and an error at token 300 arrives after the answer is already being read.
| Consumer | Stream? | Why |
|---|---|---|
| Voice conversation | Always | Silence is a failure state |
| Chat interface | Yes | Perceived latency dominates |
| Tool arguments | No | Nothing is valid until complete |
| Structured output for a system | No | Partial JSON helps nobody |
| Anything behind a hard content gate | No | A streamed token cannot be un-said |
Overlap, not speed
Three overlaps carry most of the benefit in a conversational pipeline, and none of them involves changing the model.
- Speculative context assembly - start retrieval on a stable partial transcript rather than waiting for the final one.
- Early generation with cancellation - begin generating against the partial input, and cancel cleanly if it turns out to be wrong. Only worth doing if the cancellation path is genuinely complete.
- Clause-level synthesis - send the first clause to TTS while the rest of the answer is still being generated, so first-audio latency stops depending on answer length.
Event-driven agents
A realtime agent is better modelled as an event loop than as a request handler. Audio frames, transcript revisions, tool results, interruptions and timeouts all arrive asynchronously, and any of them can invalidate work already in flight. Systems built as request/response acquire this structure eventually - usually by accident, and usually with the cancellation semantics missing.
Open questions
- How much speculative work is economically sensible? Speculation trades tokens for milliseconds and the exchange rate depends on volume.
- What is the right way to represent a conversation state that is repeatedly revised by transcript corrections?
- Do speech-to-speech models remove the need for most of this analysis, or relocate it into places we cannot inspect?