Managing context in long conversations

Context windows have grown to the point where "just put everything in" is technically possible for most conversations. It remains a bad idea, for reasons that are about attention and cost rather than about capacity.

Three tiers, three lifetimes

The important property is that these have different *lifetimes*, not just different sizes. Recent turns expire. Summaries persist for the conversation. Retrieved material is fetched fresh, which means it is never stale - the most underrated advantage of retrieval over stuffing.

What compaction destroys

Summarising a conversation is lossy in a specific and predictable way: it preserves what happened and discards what was ruled out. "The customer wants a morning appointment" survives; "the customer rejected Tuesday because of a school run" does not. Twenty turns later the agent offers Tuesday morning.

interface EpisodeSummary {
  established: string[]   // facts the agent may rely on
  rejected:    string[]   // options ruled out, and why - the part narrative loses
  decided:     string[]   // commitments made, with references
  pending:     string[]   // what is still open
  // Verbatim quotes for anything the agent committed to out loud. A
  // paraphrased promise is a promise you cannot honour precisely.
  commitments: { text: string; turn: number }[]
}

Retrieval inside a conversation is different

Retrieval in a chat interface can afford a round trip. Retrieval inside a phone call cannot: it sits directly in the caller-visible latency path. That changes the design - we retrieve speculatively on the partial transcript, cache aggressively per conversation, and prefer a slightly stale hit over a fresh miss.

It also changes what is worth retrieving. In a voice turn there is room for two or three chunks in the answer, not fifteen. Precision matters far more than recall, which inverts the usual tuning advice.

The instruction-drift problem

As a conversation grows, the system instructions become a smaller fraction of the context and their influence measurably weakens. Long conversations drift: the agent becomes more agreeable, more verbose, and more willing to do things it was told not to.

• Re-state hard constraints close to the end of the assembled context, not only at the top.
• Keep guardrails outside the prompt where possible - a check that runs on the output does not drift.
• Treat conversation length as a risk signal in its own right; the twentieth turn deserves more scrutiny than the second.

What we do not know

We do not have a principled rule for when to compact. Ours is threshold-based and hand-tuned, and it is certainly compacting too early in some conversations and too late in others. A signal based on whether recent turns actually reference older ones would be better; we have not built it.