Engineering

Designing AI agent memory

The interesting question is not how an agent remembers. It is what it is entitled to remember, and where that record lives when someone asks you to delete it.

NEOB Engineering Published Updated 9 min read
agent-memoryarchitectureprivacyomnichannel

Memory is usually introduced as a capability - the agent should remember returning customers. It is more usefully treated as a retention policy that happens to improve conversations.

Three tiers with different rules

  1. Turn state. What is happening now. Lives in the session, dies with it, never crosses a channel boundary.
  2. Episode summary. A compact record of one completed conversation. Persisted, and shared across channels only above an identity-confidence threshold.
  3. Durable facts. Preferences, entitlements, history. Persisted in a system of record and read through a tool - never in an agent-private store.
The rule that matters most

The agent should have no private long-term memory. Durable facts belong in the CRM, the booking system, the ticket system. When a customer exercises a deletion right, there is then exactly one place to delete from - a property that is close to impossible to retrofit.

Recall is gated by identity, not by availability

Having a record and being entitled to use it are different questions. A caller ID is evidence, not authentication: numbers are spoofed, phones are shared, and businesses answer from a switchboard.

We score identity evidence and gate recall on it. Below the threshold the agent may verify but not volunteer - it can ask whether this is about the enquiry from Monday once given a reason to, but it may not open the call by reciting what it knows. Confident recall from an unauthenticated channel reads as surveillance even when it is correct.

ChannelEvidenceStrength
Authenticated web sessionLoginStrong
Verified callbackWe dialled a stored numberGood
Inbound callCaller IDWeak - spoofable, shared
Web widgetCookie / deviceWeak - shared devices
MessagingAccount handleModerate
Evidence quality by channel. Only the first row is authentication.

What to write, and when

Writing memory at the end of a conversation is simple and loses the calls that end abruptly - which are disproportionately the ones worth remembering. Writing continuously is expensive and captures noise. We write on commitment events: when the agent promises something, books something, or establishes a fact it will be held to.

This has a pleasant side effect: the memory record is a list of things the business is on the hook for, which is closer to what an operations team actually wants than a transcript summary is.

Forgetting is a feature

  • Every memory entry carries a retention class and an expiry. An entry with no expiry is a bug.
  • Provenance travels with the entry: which conversation, which channel, which evidence supported it.
  • Deletion is by subject, not by conversation - the request is "forget me", not "forget that call".
  • Stale beats wrong: an expired preference should disappear rather than be applied to a customer who has changed their mind.

Where we are unsure

The evidence weights are our judgement, not a calibration - we have not measured how often each signal is actually right, and the thresholds encode a risk appetite that should probably differ by industry. A booking agent for a hair salon and one for a medical practice should not use the same numbers, and currently ours do.