Ology

Memory Is a Context-Management Problem

AI agent memory and context management: architectures, retrieval, context windows, personalization, evaluation, implementation trade-offs, reliability, privacy, and open questions.

5 sessions · 16 readings · 55 views

Curated by Rajeev Chhajer

The Context Is the Agentsession 1What Deserves to Survive?session 2Retrieval Is a Decision, Not a Lookupsession 3Long-Horizon Work Needs an External Statesession 4Can Memory Be Trusted?session 5

Opening

AI agents are becoming less like chat interfaces and more like persistent workers. They browse, call tools, modify files, query databases, and continue tasks across multiple sessions. That shift makes memory unavoidable: an agent that cannot preserve the right constraints, discoveries, and decisions is not merely forgetful—it is unreliable.

Recent systems have moved beyond the simple “store a summary in a vector database” pattern. Anthropic describes compaction, structured note-taking, just-in-time retrieval, and sub-agent architectures as different ways to manage a finite attention budget. OpenAI’s newer systems treat memory synthesis, tool history, filesystem state, institutional knowledge, and runtime inspection as distinct layers. Letta pushes the stronger claim that memory is not a detachable feature at all, but a responsibility of the agent harness.

The central difficulty is that remembering more is not necessarily better. Irrelevant history can degrade performance; summaries can erase details that later become important; persistent memories can become stale, contradictory, or poisoned; and personalization raises difficult questions about ownership, privacy, and correction. The system must decide not only what to store, but what to expose, when to update it, and how to know whether it helped.

This seminar follows memory from context design to long-term operation: what counts as memory, how memories are formed and revised, how agents navigate large stores and tool histories, how long-running systems preserve state, and how we evaluate whether continuity improves rather than quietly damages behavior.

Five questions worth arguing about

  1. 1The Context Is the Agent
  2. 2What Deserves to Survive?
  3. 3Retrieval Is a Decision, Not a Lookup
  4. 4Long-Horizon Work Needs an External State
  5. 5Can Memory Be Trusted?

The Sessions

1Session 1start here

The Context Is the Agent

The problem

The first question is conceptual: is agent memory a database feature, or is it the architecture that determines what the model can see and change? The answer matters because retrieval is only one operation. A memory system also decides how prompts are rewritten, which tools and files are visible, what survives compaction, how interaction history is represented, and whether the agent can modify its own persistent state.

Anthropic’s account starts from a finite “attention budget.” Its distinction between prompt engineering and context engineering reframes the task as continuous selection among system instructions, tool schemas, retrieved information, message history, and external artifacts. Letta approaches the same problem through an “LLM operating system” metaphor, separating kernel-managed context, user messages, memory blocks, files, and tools. Its more provocative argument is that memory cannot be bolted onto an otherwise stateless harness.

These readings belong together because they challenge the usual short- versus long-term-memory taxonomy. The important unit is not simply where information is stored, but how the harness turns external state into usable working context. That sets up the next question: once memory is treated as an active system, how should it write and revise itself?

2Session 2What Deserves to Survive?3 readings3Session 3Retrieval Is a Decision, Not a Lookup3 readings4Session 4Long-Horizon Work Needs an External State3 readings5Session 5Can Memory Be Trusted?4 readings