Ology

What Should an Agent Remember?

AI agent memory systems, including information storage, retrieval, freshness, and shared-memory trade-offs beyond context windows.

5 sessions · 13 readings · 2 views

Curated by Ben Lorenzo

A Memory Is Not a Transcriptsession 1The Write Path Is a Policysession 2What Deserves Retrieval?session 3Continuity Across Context Resetssession 4One Store, Many Principalssession 5

Opening

As agents move from answering one prompt to working across days, projects, and teams, memory is becoming an architectural choice—not just a longer transcript. A larger context window can hold more, but it does not decide what matters, what has gone stale, or what should be visible on the next turn.

Recent systems make different bets. Some synthesize conversation history in the background; others let agents revise structured memory or version files as they work. Long-running coding and research workflows also blur the line between memory and task state: a note about a user, a project plan, a correction, and a successful procedure may all persist, but they should not necessarily be stored or retrieved the same way.

Memory quality is harder to measure than recall. Retrieving a once-true fact can be worse than retrieving nothing; preserving everything can bury the useful detail; and a memory that helps one user may be inappropriate in a shared workspace. Benchmarks and deployment reports offer clues, but they measure different things—from finding a fact in conversation history to keeping a complex task on track.

This seminar traces the choices beneath those systems: what to preserve, how to revise it, what to surface, and how to carry it across context boundaries. Its central problem is how to make accumulated experience useful without letting stale, irrelevant, or unauthorized context govern what an agent does next.

Five questions worth arguing about

  1. 1Should agents preserve episodes, distilled facts, or learned procedures when each supports a different kind of future help?
  2. 2Should memory be updated immediately by the active agent, or asynchronously synthesized from a longer history?
  3. 3When recency, semantic similarity, and importance disagree, which should control what an agent brings back—and how should it know?
  4. 4When tasks outlive a context window, should continuity come from summaries, external notes, or an agent-managed workspace?
  5. 5How can shared memory accumulate institutional knowledge without exposing private context to the wrong people or agents?

The Sessions

1Session 1start here

A Memory Is Not a Transcript

The problem

A transcript preserves what was said; a memory system must decide what kind of thing it is preserving. LangChain’s taxonomy separates semantic facts, episodic experiences, and procedural know-how, while Letta’s memory blocks make some selected information editable and persistently present in the agent’s context. These are useful design distinctions, not settled categories: a correction about a user’s preference might also encode an episode, and a repeated successful workflow might become a procedure.

The tension is practical. Different representations invite different write and retrieval rules, and a system that treats every past interaction as the same kind of memory risks either losing useful structure or overfitting to a taxonomy. Once we ask what belongs in memory, the next question is who—or what—gets to change it.

Readings

  1. Memory for agentsLangChain Team
    Open ↗
  2. Memory Blocks: The Key to Agentic Context ManagementLetta
    Open ↗
2Session 2The Write Path Is a Policy2 readings3Session 3What Deserves Retrieval?3 readings4Session 4Continuity Across Context Resets3 readings5Session 5One Store, Many Principals3 readings

Next Up