Engineering

Evaluating voice agent quality

Word error rate tells you about the transcriber. It tells you almost nothing about whether the caller got what they rang for.

NEOB Engineering Published 7 min read
voice-aievaluationquality

Voice agents are evaluated with the metrics that are easy to compute, which are mostly metrics of the components rather than of the system. A pipeline can have excellent transcription, natural synthesis and a capable model, and still fail most calls.

Four levels, increasing usefulness

LevelExample metricTells you
ComponentWord error rate, TTS MOSWhether a part works. Necessary, not sufficient.
TurnResponse latency, interruption rateWhether the interaction feels right
TaskCompletion without handover, correct booking rateWhether the agent did the job
OutcomeRepeat contact within 7 days, complaint rateWhether the business is better off

Most teams measure level one and report it as though it were level three. The gap between them is where every unpleasant surprise lives.

Task completion needs a definition per deployment

A booking line completes when an appointment exists in the calendar. A support line completes when the caller stops calling. An intake line completes when a record contains the fields a human needs. These are not variants of one metric - they are different metrics that happen to share a name.

Define it per deployment, in code, before launch. Retrofitting a completion definition onto historical calls produces a number that reflects the definition more than the calls.

Handover is a success, not a failure

A handover to a human at the right moment with the context attached is the system working. Counting all handovers as failures produces a metric that improves when the agent starts stubbornly refusing to escalate - a genuinely worse product.

  • Measure late handovers: the caller asked for a person more than once first.
  • Measure cold handovers: the human received no context and started over.
  • Do not measure handover rate on its own; it is not interpretable without those two.

Metrics that mislead

  • Average call duration. Falls when the agent is efficient and also when it is useless. Undirected on its own.
  • Containment rate. Rewards refusing to escalate.
  • Model-graded answer quality. Useful for regression testing against fixed cases; unreliable as a production signal, and confidently wrong where the agent is.
  • Customer satisfaction surveys after voice calls. Response rates are low enough and skewed enough that small changes are indistinguishable from noise.

What we actually do

A fixed scenario suite run against every change - scripted calls with known correct outcomes, checked mechanically. That catches regressions. For everything else, structural signals route real conversations to a human review queue, and people judge them. It is less impressive than an automated quality score and we trust it considerably more.

On publishing numbers

We do not publish accuracy or completion rates for our own systems, because the ones we have are computed on small, German-language, industry-specific call sets. They are useful internally and would not survive being generalised. Where we cite a figure - such as bitpull.ai's sub-500 ms response target - we say whose figure it is.