Lab note

How AI agents manage context

The context window is assembled fresh every turn. What goes in it is a design decision, not a side effect.

NEOB Research Published 5 min read

A common misconception is that an agent "has" a conversation in the way a person does. It does not. Every turn, a context window is assembled from scratch and handed to a model that has no memory of the previous call.

Instructions

  • role
  • constraints
  • tone
  • escalation rules

Tools

  • available tool schemas

Retrieved knowledge

  • relevant chunks for this turn

Conversation

  • recent turns verbatim
  • earlier turns compacted

Current input

  • what the user just said
A typical assembled context, in priority order.

Why bigger windows did not solve this

Three reasons. Cost scales with what you put in. Attention is not uniform across a long context, so buried instructions lose influence. And a large window filled indiscriminately contains contradictions - an old answer that has since been superseded is still, as far as the model can tell, part of the record.

Compaction and what it costs

Compaction replaces older turns with a summary. It is lossy in a specific direction: it keeps what happened and drops what was ruled out. That is why an agent twenty turns into a conversation will cheerfully re-offer something the customer already rejected.

Summarising toward a constraint set - established, rejected, decided, pending - rather than toward a narrative preserves exactly the material later turns need. It is a small change with a disproportionate effect.

Retrieval is context management

Retrieval is often filed under knowledge, but its function is contextual: it puts a small, relevant, *current* slice of a large corpus into the window instead of a large stale one. Freshness is the underrated half of that.