Conventional monitoring asks whether the system responded. Agent monitoring has to ask whether the conversation went anywhere - a question with no status code.
Trace the loop, not the request
The unit of observation is the conversation, and inside it the loop iteration. A useful trace lets you replay what the agent saw and why it chose what it chose.
- Per iteration: assembled context size, retrieval hits, tool considered, tool chosen, arguments, result classification, latency per stage.
- Per conversation: turn count, tool calls, interruptions, handovers, resolution signal, total cost.
- Per deployment: distribution of the above - the tail is where the failures live, and averages hide it completely.
Signals that predict a bad conversation
| Signal | What it usually means |
|---|---|
| Repeated identical tool call | Empty result being reported as an error |
| Turn count above the deployment norm | The agent is not converging |
| Repeated user rephrasing | It is not understanding, and the user knows it |
| Rising interruption rate | Answers too long, or endpointing mistuned |
| Retrieval returning nothing | Knowledge gap - often a document that failed to index |
| Handover immediately after a specific topic | A capability gap with a known name |
None of these require judging answer quality, which is the expensive part. They are structural, cheap to compute, and they point at the conversations a human should actually look at.
Sampling and privacy
Full transcripts are the most useful and most sensitive artefact in the system. We keep structural traces broadly and content narrowly: retention on transcripts is short, access is logged, and the review queue is built from the structural signals rather than from browsing conversations.
Redacting transcripts for observability tends to remove exactly the part you needed - the customer's actual phrasing. Decide deliberately whether a given trace is for debugging behaviour (needs content) or for operating the system (does not), and store them under different rules.
Alert on the derivative
Absolute thresholds age badly across deployments - a handover rate that is alarming for a booking line is normal for technical support. Alert on change instead: today's distribution against the trailing baseline for that deployment. A model update, a prompt change or a newly indexed document usually shows up as a step, not as a threshold breach.
What we have not solved
Automated quality judgement. Model-graded evaluation is useful for regression testing against fixed cases and, in our experience, unreliable as a production quality signal - it drifts, and it is confidently wrong in the same places the agent is. We use structural signals to *route* conversations to people, and let people judge.