How AI Agents Use Memory Across Sessions

Memory Types in Agent Systems

Working memory is the information the agent actively uses during the current task. It includes the conversation history, tool results, intermediate computations, and the current task state. Working memory lives in the context window and is directly accessible to the model on every turn. The limitation is size: working memory cannot exceed the context window, and filling the window with historical data leaves less space for current reasoning.

Short-term memory persists within a single session but is too large to keep entirely in the context window. When a conversation generates more content than the window can hold, older messages move to short-term memory storage. The agent can retrieve specific items from short-term memory when needed, rather than keeping everything in the active context. This is implemented through summarization (condensing older messages into shorter summaries) or selective retrieval (loading specific past messages on demand).

Long-term memory persists across sessions indefinitely. It stores facts, preferences, past interactions, learned procedures, and accumulated knowledge in external storage systems. Long-term memory is what makes an agent feel like it "knows" you: it remembers your name, your preferences, your project context, and the decisions you made in previous conversations. Without long-term memory, the agent treats every session as a first meeting.

Episodic memory records complete interaction sequences as discrete episodes. Rather than extracting individual facts from past conversations, episodic memory preserves the full context of what happened: the task, the approach taken, the tools used, the results obtained, and whether the outcome was successful. When the agent encounters a similar situation, it can retrieve the relevant episode and use it as a guide, essentially learning from its own experience.

Vector Database Memory

Vector databases are the most common infrastructure for agent long-term memory. They store information as numerical vectors (embeddings) in a high-dimensional space where semantically similar information is located close together. When the agent needs to recall relevant past information, it converts its current question into an embedding and searches for the nearest stored embeddings.

The embedding process converts text into a fixed-length numerical vector using an embedding model. The same text always produces the same embedding, and semantically similar texts produce similar embeddings. "How do I reset my password?" and "I forgot my login credentials" produce embeddings that are close together in vector space, even though they share few words. This semantic similarity is what makes vector search useful for agent memory: the agent can find relevant past interactions even when the current question is phrased differently from the stored information.

Retrieval quality depends on several factors. The embedding model determines how well semantic similarity is captured. The chunk size (how much text is stored per embedding) affects precision: small chunks are more precise but may miss context, while large chunks capture more context but may include irrelevant information. The number of results returned (top-k) affects recall: returning more results increases the chance of finding the relevant information but also increases the amount of irrelevant content the agent must process.

Metadata filtering enhances retrieval accuracy by narrowing the search space before vector similarity is computed. Each stored memory can include metadata like the timestamp, the user ID, the topic category, the session ID, and a relevance score. When the agent searches for memories about a specific user or topic, metadata filters eliminate irrelevant memories before the similarity search runs, improving both speed and accuracy.

Structured Knowledge Stores

Not all agent memory fits the vector search paradigm. User preferences, configuration settings, and factual knowledge are better stored in structured formats (key-value stores, relational databases, or JSON documents) that support exact lookups. When the agent needs to know a user preferred language or their account status, an exact database query is faster and more reliable than a semantic similarity search.

Knowledge graphs represent relationships between entities in a structured format. A knowledge graph might store that User A manages Project B, Project B uses Technology C, and Technology C has a known issue with Version D. When the agent encounters a question about Project B, it can traverse the knowledge graph to find related entities and their relationships, providing richer context than a flat memory store.

Hybrid memory systems combine vector search for fuzzy, semantic retrieval with structured stores for exact, factual retrieval. The agent runtime queries both systems in parallel and merges the results. This hybrid approach provides the flexibility of semantic search for open-ended questions and the precision of structured queries for specific facts.

Memory Management

Memory systems require ongoing management to remain useful. Without management, memories accumulate indefinitely, consuming storage and degrading retrieval quality as irrelevant old memories outnumber relevant current ones.

Memory decay gradually reduces the importance of older memories. Memories that have not been accessed recently or that relate to obsolete contexts lose priority in retrieval results. This prevents the agent from being influenced by outdated information that no longer reflects current reality. The decay rate should be tunable per memory type: user preferences might decay slowly (preferences are relatively stable), while task-specific memories might decay quickly (specific task contexts become irrelevant once the task is complete).

Memory consolidation merges multiple related memories into a single, more comprehensive memory. If the agent has stored five separate memories about a user project, consolidation combines them into one comprehensive project summary. This reduces storage requirements and improves retrieval quality because the consolidated memory contains all relevant information in one place rather than scattered across multiple entries.

Conflict resolution handles situations where stored memories contradict each other. The user might have expressed one preference in January and a different preference in March. A product might have had one set of features in an earlier version and different features in the current version. Conflict resolution policies (most recent wins, highest confidence wins, ask the user to clarify) ensure that the agent uses accurate, current information rather than outdated or conflicting data.

Privacy and Access Control in Memory Systems

Agent memory systems often store sensitive information: personal data, business confidential details, proprietary strategies, and private communications. Access control ensures that memories stored from one user interaction are not leaked to another user, and that memories from one organization are not accessible to agents serving a different organization.

Tenant isolation separates memory stores by user, organization, or deployment environment. Each tenant has its own memory namespace, and queries can only access memories within the current tenant scope. This isolation is enforced at the infrastructure level (separate database partitions or collections) rather than relying on the agent to filter appropriately, because a model-level filter can be bypassed by prompt injection or reasoning errors.

Retention policies define how long memories are kept and when they must be deleted. Regulatory requirements like GDPR mandate that personal data be deleted upon request. Business policies might require deletion of memories from departed employees or terminated contracts. Automated retention enforcement ensures that these policies are applied consistently without relying on manual cleanup, which is inevitably inconsistent and incomplete.

Memory auditing tracks who stored what, when it was accessed, and by which agent. This audit trail is essential for compliance, debugging, and security incident investigation. When a user questions how the agent knew a particular fact, the audit trail shows exactly which memory was retrieved and when it was originally stored. When a security incident occurs, the audit trail reveals which memories were accessed and whether any unauthorized access took place.