Conventional monitoring answers "did the system respond". Agent monitoring has to answer "did the conversation go anywhere", and there is no status code for that.
The unit is the conversation
A trace should let you replay a decision: what context was assembled, what was retrieved, which tools were available, which was chosen, with what arguments, and what came back. Without that, debugging is guessing with extra steps.
Cheap signals that find bad conversations
- The same tool called repeatedly with the same arguments - usually an empty result mis-typed as an error.
- Turn count well above the norm for that deployment - the agent is not converging.
- The user rephrasing repeatedly - it is not understanding, and they know it.
- Retrieval returning nothing - often a document that silently failed to index.
- Handover clustered around one topic - a capability gap you can name.
None of these requires judging answer quality. They are structural, nearly free to compute, and they are how you decide which conversations a person should actually read.
Alert on change, not on thresholds
A handover rate that is alarming for a booking line is normal for technical support. Compare each deployment against its own trailing baseline; model updates, prompt edits and newly indexed documents show up as steps, not as threshold breaches.