Lab note

From prototype to production AI agent

The demo is roughly ten per cent of the work, and it is not the ten per cent that determines whether the thing survives contact with customers.

NEOB Engineering Published 9 Jul 20266 min read

Agent prototypes are unusually convincing. They work on the happy path immediately, which makes the remaining distance look shorter than it is.

Concern	Prototype	Production
Tool errors	Throws, loop retries	Error / empty / refusal distinguished at the boundary
Side effects	Fires every time	Idempotency keys derived from intent
Permissions	Full access	Scoped per tenant and per agent
Irreversible actions	Just does them	Human approval gate
Failure path	Apologises	Handover with context attached
Observability	Console logs	Per-iteration traces, structural signals, review queue
Cost	Unbounded	Per-conversation budget and a hard stop
Change safety	Try it and see	Fixed scenario suite run on every change

The same capability, at two levels of seriousness.

The three that get skipped

Idempotency. Agents retry. Without keys derived from intent, retries duplicate side effects, and the duplicate is discovered by the customer.
The handover path. Every agent needs a way to stop and fetch a person, carrying context. Retrofitting this is painful because the context was never structured for handing over.
A cost ceiling. A loop that cannot terminate is a bug in the prototype and an invoice in production.

What we would tell ourselves earlier

Instrument the stages separately before optimising any of them, and decide what "task completed" means for this deployment *before* launch. Both are cheap in week one and expensive in month six - the second one especially, because a completion definition retrofitted onto historical calls tells you about the definition, not the calls.