Barge-in without false interruptions

Problem

A voice agent that cannot be interrupted is exhausting. A voice agent that stops at every sound is worse - it produces a stuttering half-turn where neither party knows who is speaking.

The failure cases that mattered in real calls were not the obvious ones. They were: a caller saying "mhm" while listening; a second person talking in the same room; a car radio; and the agent hearing its own audio through a speakerphone.

Hypothesis

Energy-based voice activity detection alone cannot separate an interruption from a backchannel. Adding a short semantic check on the first fragment of interrupting speech is enough to cut false stops without adding perceptible delay.

Architecture

The pipeline runs two detectors in parallel while the agent is speaking. A fast one decides that *something* is being said; a slower one decides whether it was addressed to the agent and whether it changes anything.

The important design choice is that detection has three outcomes, not two. Between "keep talking" and "stop" sits ducking: lower the agent volume, keep the audio queue, and wait ~300 ms to see whether the caller continues. Most backchannels resolve inside that window and the agent recovers without a visible seam.

Implementation

The backchannel classifier is deliberately small. It runs on the first partial transcript fragment and only asks one question: is this a continuer ("mhm", "yes", "okay", "genau") or is it content? A list plus a length check gets most of the way; the model is only consulted when the fragment is ambiguous and long enough to matter.

type Decision = 'ignore' | 'duck' | 'stop'

function onSpeechWhileSpeaking(ev: SpeechEvent): Decision {
  // Echo of our own output. Handled before anything else - on speakerphone
  // this was by far the most common false interruption.
  if (ev.echoCorrelation > ECHO_THRESHOLD) return 'ignore'

  // Too short to be an interruption at all.
  if (ev.durationMs < MIN_SPEECH_MS) return 'ignore'

  // Continuers: the caller is listening, not taking the floor.
  if (isBackchannel(ev.partialText)) return 'duck'

  return 'stop'
}

On stop, three things have to happen together, and getting only two of them right is what produces the classic "agent answers the question it was already answering" bug: cancel the TTS stream, drop the queued clauses produced by clause-level streaming, and truncate the assistant turn in the conversation state to what was actually played out loud.

Result

Ducking as a middle state turned out to carry most of the benefit. Once "mhm" stopped ending the agent turn, the remaining false stops were dominated by room noise and second speakers - a different problem, addressed by echo cancellation and gating rather than by classification.

The second result was structural: interruption handling belongs to the conversation-state layer, not the audio layer. We had originally built it as an audio concern, and every bug we chased came from state and audio disagreeing about what the agent had said.

Limitations

What this experiment does not establish. Listed because an experiment without limitations is an advertisement.

• The backchannel list is language-specific. It works for German and English; every additional language needs its own list, and code-switching mid-call defeats it.
• Two speakers in one room remains unsolved here. We gate rather than separate, which means the agent occasionally stops for a conversation it is not part of.
• Ducking adds a perceptible volume dip. Some callers read it as a connection problem.
• We report this qualitatively. We do not have a false-interruption rate we would be willing to publish as a number, because our labelled call set is too small and too German to generalise.

Next steps

• Replace the continuer list with a small classifier trained on labelled call audio, so new languages do not need hand-written lists.
• Speaker diarisation on the inbound leg to distinguish the caller from the room.
• Measure how long a duck can last before callers interpret it as a dropped line.