Where the milliseconds go in a phone call

Problem

A voice agent is judged in the gap between the caller finishing a sentence and the agent starting to speak. Everything else - the quality of the answer, the voice, the integrations - is only noticed if that gap is short enough to be forgivable.

The usual reflex when that gap is too long is to reach for a faster model. That is almost always the wrong lever. In a telephone pipeline the model is one of six or seven stages, and several of the others are larger, more variable, and much cheaper to fix.

The problem we set ourselves: produce a stage-by-stage account of a single turn that is precise enough to argue with, so optimisation work can be aimed rather than guessed.

Hypothesis

Perceived response time in a telephone voice agent is dominated by endpointing and first-token/first-audio latency, not by total model throughput. Therefore overlapping stages beats speeding any single one of them up.

Architecture

The pipeline under test is a cascaded one: audio arrives over a telephony leg, is transcribed incrementally, handed to a language model that may call tools, and the reply is synthesised and streamed back.

Two stages in that chain are not compute at all. Transport is the network path: for a PSTN call this includes the carrier leg, the SIP gateway and the jitter buffer at each end. Endpointing is the decision that the caller has actually finished, rather than paused. Both are structural, and both are frequently larger than the model call.

Read naively, that sums to something a caller would notice. The point of the exercise is that it does not have to be summed: four of those seven stages can be made to run inside another one.

Implementation

Overlap 1 - speculative context assembly

Context assembly does not need the final transcript. Once the incremental transcript is stable enough - in practice, once the last two hypotheses agree on everything but the final word - retrieval and state assembly can start against the partial text. If the final transcript differs materially, the work is discarded. It usually does not.

Overlap 2 - early model start with cancellation

The same trick applies one stage later, and is riskier: start generating against the partial transcript, and cancel if endpointing resolves differently. This is only worth doing when generation is cheap to abandon and the cancellation path is genuinely wired up. Half-implemented cancellation produces the worst possible failure - two overlapping replies.

Overlap 3 - streaming synthesis at clause boundaries

The biggest single win. Do not wait for the full model output before synthesising. Emit the first clause to TTS as soon as the model produces a clause boundary, and stream audio while the rest of the sentence is still being generated. First-audio latency stops being a function of answer length.

// Cut the model stream at clause boundaries and hand each piece to TTS
// immediately. First audio then depends on the first clause only, not on
// how long the full answer turns out to be.
const BOUNDARY = /[.!?…,;:]\s|\n/

async function* clauses(tokens: AsyncIterable<string>) {
  let buf = ''
  for await (const t of tokens) {
    buf += t
    const m = BOUNDARY.exec(buf)
    if (m && buf.length > MIN_CLAUSE) {
      const cut = m.index + m[0].length
      yield buf.slice(0, cut)
      buf = buf.slice(cut)
    }
  }
  if (buf.trim()) yield buf
}

for await (const clause of clauses(model.stream())) {
  // Backpressure matters: if the caller barges in, this queue is what
  // has to be dropped, not just the current chunk.
  await tts.push(clause)
}

The stage we could not overlap

Endpointing is irreducible, because it is a decision about the future: has the caller finished, or are they thinking? A shorter silence threshold cuts latency and increases interruptions. This is a tuning trade-off, not an engineering one, and it is the reason the same agent feels different on a mobile call than on a landline.

Result

What came out of this is a design rule rather than a number: treat the turn as a pipeline to be overlapped, not a sum to be shortened. With clause-level streaming in place, the caller-visible latency is roughly transport + endpointing + first token + first audio chunk - the answer length drops out of the equation entirely.

The production system this fed into is bitpull.ai, whose published figure for the caller-visible gap is under 500 ms. That is their stated product figure and the target this budget was built to serve; we are not presenting it here as an independent measurement.

The second outcome was a change in how we debug. Because the stages are instrumented separately, a slow call now produces a stage breakdown rather than a single total, and the answer to "why was that call sluggish" is usually visible in one line of the trace.

Limitations

What this experiment does not establish. Listed because an experiment without limitations is an advertisement.

• The budget above is a design target. We publish it as a shape to argue with, not as a benchmark - per-call variance from carrier routing and model load is larger than several of the individual stages.
• Numbers come from a cascaded pipeline. A speech-to-speech model changes the stage list entirely and invalidates most of this decomposition.
• Speculative work costs money. Starting generation on a partial transcript that gets revised means paying for tokens that are thrown away; at low call volume that is irrelevant, at high volume it is a real line item.
• Endpointing tuning is per-deployment. A threshold that works for a reception desk is wrong for a technical support line where callers pause to read serial numbers.

Next steps

• Compare this cascade against a speech-to-speech model on the same telephony leg, holding transport constant.
• Semantic endpointing: use the partial transcript to predict whether a sentence is complete, instead of relying on a silence timer alone.
• Publish a reproducible harness so the stage breakdown can be run against other stacks rather than argued about.