The Memory Layer: Persistent Agent Knowledge

Types of Agent Memory

Agent memory operates across three timescales, each requiring different implementation strategies. Working memory is the current conversation context: the messages exchanged in the active session that the model uses to maintain coherence. This is handled automatically by the LLM's context window and requires no special infrastructure. The limitation is that context windows are finite (typically 8K to 128K tokens), and once a conversation exceeds that limit, older messages must be dropped or summarized.

Short-term memory persists across sessions for a specific user or project. When a user returns to an agent the next day, short-term memory provides continuity: the agent knows what was discussed previously, what decisions were made, and what the user was working on. This is typically implemented as stored conversation history loaded into the prompt at the start of each new session, with the most recent or most relevant exchanges prioritized.

Long-term memory captures durable facts, preferences, and knowledge that should persist indefinitely. A user's name, role, communication preferences, project architecture decisions, frequently asked questions, and learned domain knowledge all belong in long-term memory. Unlike conversation history (which is chronological), long-term memory is organized by topic and relevance, retrieved selectively rather than loaded wholesale.

Conversation History Storage

The simplest and most essential form of memory is storing conversation history in a database. Every message sent to the agent and every response generated gets stored with a timestamp, session identifier, and user identifier. When a user starts a new session, recent history is loaded from the database and included in the system prompt, giving the model context about previous interactions.

PostgreSQL is the standard choice for conversation storage because it handles concurrent access, provides ACID transactions, supports efficient querying by user and session, and integrates with your existing infrastructure. SQLite works well for single-user applications or development environments where the operational overhead of a separate database server is unnecessary.

The challenge with raw conversation history is that it grows without bound and including all of it in every prompt wastes tokens and degrades model attention. A conversation history of 50 exchanges contains a few important decisions buried in a lot of routine back-and-forth. Effective memory management means extracting the important information and presenting it efficiently rather than dumping everything into the context.

Summarization-Based Memory

Summarization addresses the unbounded growth problem by periodically condensing conversation history into compact knowledge statements. After every 10 or 20 exchanges, the system asks the LLM to summarize the key information from those messages: what was discussed, what was decided, what facts were learned. These summaries are stored as long-term memory entries, and the raw conversation history can be archived or discarded.

The summaries are typically embedded in the same vector database used for document RAG. When a new conversation starts, the system embeds the initial user message, searches the memory database for relevant summaries, and includes them in the prompt context. This retrieval-based approach means the agent only remembers information relevant to the current query, keeping prompt sizes manageable regardless of how much total memory has accumulated.

MemGPT (now called Letta) pioneered a more sophisticated approach where the agent itself manages its own memory. The agent has explicit tools for writing to memory, reading from memory, and searching memory. It decides what to remember and what to forget based on its assessment of importance. This self-managed approach produces more intelligent memory behavior but requires a capable model (13B or larger) and careful prompt engineering to prevent the agent from either hoarding trivial information or discarding important context.

Knowledge Graphs for Structured Memory

Knowledge graphs store information as entities and relationships rather than flat text. An entity might be a person ("Alice, senior developer"), a project ("payment service rewrite"), or a concept ("the team uses PostgreSQL for everything"). Relationships connect entities: "Alice leads the payment service rewrite," "the payment service depends on the API gateway." When the agent needs context about a topic, a graph query retrieves the entity and all its connected information.

The advantage of graph-based memory over vector-based memory is precision in multi-hop reasoning. If a user asks "who is working on the project that depends on the API gateway," a graph query follows the relationship chain (API gateway -> payment service -> Alice) to find the answer. Vector similarity search might not connect these concepts if they were never mentioned in the same passage.

Neo4j is the established choice for knowledge graph storage, offering a mature query language (Cypher), strong visualization tools, and extensive documentation. For smaller deployments, Apache AGE adds graph capabilities to PostgreSQL as an extension, letting you store graph data and relational data in the same database. Both options support self-hosted deployment in Docker containers.

Implementing Memory in Practice

Start with conversation history in PostgreSQL. This alone provides meaningful continuity across sessions and requires minimal implementation effort. Store each message as a row with columns for session ID, user ID, role (user or assistant), content, and timestamp. Load the most recent 10 to 20 messages from the current session at the start of each request.

Add summarization-based long-term memory when conversations regularly exceed 20 exchanges or when users expect the agent to remember information from days or weeks ago. Run summarization as a background process after each session ends, embed the summaries, and retrieve them at the start of new sessions. This can be implemented with a few hundred lines of code using your existing vector database.

Consider a knowledge graph only if your agents need to track complex relationships between many entities (people, projects, dependencies, decisions) and answer questions that require traversing those relationships. For most use cases, vector-based memory retrieval provides sufficient recall without the added complexity of graph infrastructure.

Memory Maintenance and Decay

Memory systems that only accumulate information without pruning eventually degrade in quality. Old memories become irrelevant as projects change, preferences evolve, and facts become outdated. A well-designed memory layer includes decay mechanisms that reduce the relevance score of memories over time, ensuring that recent information is prioritized over stale context. Simple approaches include reducing the weight of memories based on age, while more sophisticated systems track how often each memory is accessed and prioritize frequently-used knowledge.

Periodic memory auditing identifies contradictions and outdated information. When an agent stores a fact that contradicts something it stored months ago, both memories exist in the database. Without auditing, the agent might retrieve the outdated reference instead of the current one. Deduplication processes that compare new memories against existing ones and update or replace conflicting entries prevent this kind of knowledge decay. Running these processes as scheduled background tasks keeps the memory database accurate without manual intervention.

Storage costs also grow with unmanaged memory accumulation. Each embedded memory occupies space in the vector database, and similarity search performance can degrade as the collection grows into millions of entries. Setting retention policies that automatically archive memories older than a configurable threshold, delete memories that have never been retrieved, and consolidate related memories into single entries keeps the memory database performant and manageable. These maintenance operations should run during off-peak hours to avoid competing with active inference requests for system resources.

Privacy and Memory Isolation

In multi-user deployments, memory isolation is essential. Each user's memories must be stored and retrieved independently, with no possibility of one user's personal information appearing in another user's agent context. Implement this through strict metadata filtering on vector searches: every memory is tagged with a user identifier, and every retrieval query includes a filter that restricts results to the requesting user's memories. Test this isolation rigorously, because a memory leak between users is both a privacy violation and a trust-destroying bug.

Consider also the regulatory implications of persistent memory. If your agents serve European users, stored memories may qualify as personal data under GDPR, requiring mechanisms for data export and deletion on request. If your agents handle healthcare or financial information, stored memories may be subject to industry-specific retention and security requirements. Design your memory schema with these obligations in mind from the beginning, because retrofitting compliance into an existing memory system is significantly harder than building it in from the start.