The piece on approaches to agent memory lays out a clean taxonomy of the storage strategies available to AI agents: in-context memory, external retrieval, and structured key-value state. The taxonomy is accurate and the framing is useful. But the more I work on persistent agent systems (I maintain a Discord bot with a persistent memory layer), the more the storage taxonomy feels like the easier half of the problem. The harder half is write timing, and most of the writing on agent memory glosses over it.
In-Context Memory: The Default and Its Limits
The simplest form of agent memory is the context window itself. The LLM receives a prompt, reasons over it, and responds. For OpenAI’s GPT-4o, that ceiling is 128,000 tokens. For Claude 3.7 Sonnet, 200,000. For Gemini 1.5 Pro, 1,000,000.
Long-context models look like a solution. If you can fit tens of thousands of messages into context, you might not need an external memory system at all. Two factors limit this framing. The first is cost: context tokens are priced at inference time, on every call. The second is a subtler quality problem. The “lost in the middle” finding from Liu et al. (2023) showed that LLM recall degrades significantly for information placed in the middle of long contexts, with performance recovering only at the very start and end. Longer contexts do not give proportionally better retrieval of their contents.
The context window is working memory in the cognitive science sense: fast, directly accessible, not persistent across sessions, and limited by capacity rather than content relevance.
External Retrieval: Vector Stores and RAG
The standard extension is to move older information into an external store and retrieve relevant chunks at query time. This is the retrieval-augmented generation pattern applied to conversational state rather than documents.
In practice: embed past interactions as dense vectors, store them in something like pgvector, Chroma, or Pinecone, and retrieve the top-k most semantically similar memories at each turn. The retrieved chunks get injected into the context window before the model responds.
A simple memory retrieval call against pgvector looks roughly like this:
SELECT content, embedding <=> $1 AS distance
FROM memories
WHERE user_id = $2
ORDER BY distance
LIMIT 5;The failure mode for vector retrieval is fuzzy recall. If a user mentioned their favorite editor is Emacs, a query about “text editing preferences” will probably surface it. A query about “keyboard shortcuts” might not. The cosine similarity threshold determines precision versus recall, and there is no universally correct value. You tune it per deployment and still get it wrong in edge cases.
Structured Key-Value Storage: Exact Facts
For information that needs exact recall, vector search is the wrong tool. User preferences with specific identifiers, configuration values, domain knowledge a user has explicitly provided: these belong in a simple key-value structure retrieved by direct lookup.
The implementation is straightforward: a JSON file, a SQLite table, a Redis hash. Lookup is deterministic. Either the key exists or it does not. No threshold tuning, no embedding drift.
The limitation is that structured storage requires deciding the schema at write time. You need to extract a structured fact from the conversation and map it to a key. For unstructured conversational input, that extraction step typically involves a secondary LLM call:
System: Extract factual preferences from the conversation snippet below.
Output JSON with fields: key, value, confidence (0.0 to 1.0)
Conversation: "I prefer TypeScript over plain JavaScript for anything
beyond quick scripts."
Output: {"key": "language_preference", "value": "TypeScript over JavaScript", "confidence": 0.9}This adds latency and introduces a new failure mode: extraction errors that silently corrupt the memory store. A misclassified preference gets written with high confidence and then surfaces in the wrong context for the rest of the agent’s life.
MemGPT’s Virtual Context Model
The most architecturally interesting approach is the one introduced in the MemGPT paper from Berkeley in 2023, now productized as Letta. The core idea is to treat the LLM like a CPU and the context window like a register file, with the model itself managing what gets loaded from external storage.
In MemGPT’s model, the LLM receives tool calls it can use to explicitly write to and read from memory. When the context approaches its limit, the model can evict old content and retrieve relevant content on demand. The model drives its own memory management rather than having an external pipeline impose decisions on it.
This sidesteps the write-timing problem in one sense because the model decides. The trade-off is that this requires the model to reason about its own memory needs on every call, adding cognitive load and introducing a new class of failures: the model forgets to persist something important, with no signal to the developer that it happened.
The CoALA paper (Cognitive Architectures for Language Agents, Sumers et al. 2023) provides a useful framework for thinking about the four memory types drawn from cognitive science: working memory (in-context), episodic memory (past events), semantic memory (general facts), and procedural memory (how to accomplish tasks). MemGPT collapses the distinction between the agent and the memory system; the other approaches treat them as separate engineering concerns.
The Write-Timing Problem
Here is the piece the storage taxonomy does not address: when do you commit something to memory? The answer determines the quality of everything downstream.
Several obvious strategies each carry a distinct failure mode.
Write after every message. This is maximally complete but generates large volumes of redundant and low-signal memories. Filler messages, clarification requests, and acknowledgments flood the store and degrade retrieval quality over time.
Write at conversation end. Clean in theory, but in persistent chat environments like Discord channels, conversation boundaries are blurry. The conversation does not end; it gets quiet for a while and then resumes without any clear boundary event.
Write on LLM-detected signal. This is what MemGPT does. The model explicitly decides to persist something. The failure mode is inconsistency: the model commits some things and silently drops others, with no obvious pattern and no debuggable audit trail.
Write on a schedule via background summarization. Mem0 uses a background consolidation pass that summarizes recent interactions into structured memories. Recent information is unavailable until the consolidation runs, but the quality of what gets written tends to be higher because the summarization step compresses noise before storage.
Hybrid Approaches in Practice
In Ralph, I landed on three layers. The context window holds the current conversation. A SQLite-backed key-value store holds user-specific facts and bot state that users have explicitly provided or that the bot has extracted with high confidence. A running journal file accumulates interaction summaries and gets periodically consolidated into the retrieval store.
The thing that surprised me most was how much the summarization strategy mattered relative to the storage choice. Whether you use pgvector or Chroma makes almost no difference for a small deployment. The decisions that compound across every interaction are: what triggers a write, how granular the chunks are, and how aggressively you distill before storing.
A memory system that writes raw message text verbatim performs much worse on retrieval than one that distills each interaction into a short structured summary. The embedding quality of a filler message is always going to be poor. The embedding quality of “user confirmed preference for async/await over promise chains in their Node.js projects” is much better, and retrieval will surface it at the right time. Zep is one of the more production-ready frameworks built around this architecture: session-level context, session-to-long-term memory extraction, and fact extraction running in the background without blocking the main inference call.
The design space for agent memory is real and worth understanding, but the framework that shapes outcomes most is not the storage taxonomy. It is the pipeline that feeds the storage: what triggers a write, what gets written, and how it gets compressed before storage. Those decisions determine whether the memory layer helps the agent or accumulates noise that dilutes retrieval quality with every passing conversation.