Voice agents are evaluated with the metrics that are easy to compute, which are mostly metrics of the components rather than of the system. A pipeline can have excellent transcription, natural synthesis and a capable model, and still fail most calls.
Four levels, increasing usefulness
| Level | Example metric | Tells you |
|---|---|---|
| Component | Word error rate, TTS MOS | Whether a part works. Necessary, not sufficient. |
| Turn | Response latency, interruption rate | Whether the interaction feels right |
| Task | Completion without handover, correct booking rate | Whether the agent did the job |
| Outcome | Repeat contact within 7 days, complaint rate | Whether the business is better off |
Most teams measure level one and report it as though it were level three. The gap between them is where every unpleasant surprise lives.
Task completion needs a definition per deployment
A booking line completes when an appointment exists in the calendar. A support line completes when the caller stops calling. An intake line completes when a record contains the fields a human needs. These are not variants of one metric - they are different metrics that happen to share a name.
Define it per deployment, in code, before launch. Retrofitting a completion definition onto historical calls produces a number that reflects the definition more than the calls.
Handover is a success, not a failure
A handover to a human at the right moment with the context attached is the system working. Counting all handovers as failures produces a metric that improves when the agent starts stubbornly refusing to escalate - a genuinely worse product.
- Measure late handovers: the caller asked for a person more than once first.
- Measure cold handovers: the human received no context and started over.
- Do not measure handover rate on its own; it is not interpretable without those two.
Metrics that mislead
- Average call duration. Falls when the agent is efficient and also when it is useless. Undirected on its own.
- Containment rate. Rewards refusing to escalate.
- Model-graded answer quality. Useful for regression testing against fixed cases; unreliable as a production signal, and confidently wrong where the agent is.
- Customer satisfaction surveys after voice calls. Response rates are low enough and skewed enough that small changes are indistinguishable from noise.
What we actually do
A fixed scenario suite run against every change - scripted calls with known correct outcomes, checked mechanically. That catches regressions. For everything else, structural signals route real conversations to a human review queue, and people judge them. It is less impressive than an automated quality score and we trust it considerably more.
We do not publish accuracy or completion rates for our own systems, because the ones we have are computed on small, German-language, industry-specific call sets. They are useful internally and would not survive being generalised. Where we cite a figure - such as bitpull.ai's sub-500 ms response target - we say whose figure it is.