Vector Search: How Agents Find Relevant Memories

From Text to Vectors: The Embedding Idea

The foundation of vector search is the embedding, a way of turning a piece of text into a list of numbers that captures its meaning. An embedding model reads a sentence and outputs a vector, often several hundred to a few thousand numbers long, positioning that text as a single point in a high-dimensional space. The crucial property is that texts with similar meanings produce vectors that sit close together in that space, while unrelated texts land far apart. The phrase how do I reset my password and the phrase I forgot my login credentials map to nearby points, even though they use different words.

This is a profound shift from treating text as a string of characters to treating it as a location in a space of meaning. Once every memory is a point, finding relevant memories becomes a geometry problem: which stored points are nearest to the point representing the query. The quality of the embedding model determines how faithfully meaning is captured, which is why choosing and configuring it matters so much, a topic covered in depth in embedding models for agent memory. For now, the key idea is simply that text becomes a vector, and meaning becomes distance.

Measuring Closeness: Similarity in Vector Space

If memories are points in space, the system needs a precise way to measure how close two points are. The most common measure in agent memory is cosine similarity, which looks at the angle between two vectors rather than the straight-line distance between them. Two vectors pointing in nearly the same direction have a cosine similarity near one and are considered very similar; two pointing in unrelated directions score near zero. Cosine similarity is popular because it focuses on the direction of meaning and ignores differences in magnitude, which makes it robust across texts of different lengths.

Other measures exist, such as Euclidean distance, the ordinary straight-line distance, and the dot product, which combines direction and magnitude. The choice interacts with how the embedding model was trained, and most vector databases let you pick the measure that matches your embeddings. In practice, cosine similarity is the default for text memory because embedding models are typically trained so that the angle between vectors reflects semantic closeness. Whatever the measure, the principle is the same: convert the comparison of meanings into a single number that ranks every stored memory by how relevant it is to the query.

Searching Fast: Approximate Nearest Neighbor

Finding the closest vectors to a query sounds simple, but doing it exactly means comparing the query against every stored memory, which becomes far too slow once a store holds hundreds of thousands or millions of entries. The solution is approximate nearest neighbor search, a family of algorithms that find the closest vectors almost always correctly while examining only a small fraction of the store. They trade a tiny, usually negligible loss in accuracy for an enormous gain in speed.

The most widely used approach builds a navigable graph that links each vector to its neighbors, so a search can start anywhere and quickly hop toward the region of the space nearest the query, a structure known as a hierarchical navigable small world index. Other methods group vectors into clusters and search only the most promising clusters. The practical consequence for an agent builder is that vector databases can return the most relevant memories from a massive store in a few milliseconds, which is what makes real-time recall feasible. This indexing is one of the main things a managed vector service provides, a consideration in choosing local versus cloud memory.

Why Vector Search Beats Keyword Matching, and Where It Fails

The reason vector search has become the default for agent memory is that it matches on meaning, which is exactly what recall requires. Keyword search, the older approach, only finds memories that share literal words with the query, so it misses anything phrased differently. A user who once mentioned relocating to a new city would not be found by a keyword search for moved house, but vector search surfaces it because the meanings are close. This ability to bridge different wordings is what makes an agent feel like it genuinely understands rather than mechanically matching strings.

Vector search has real weaknesses, though, and knowing them is what keeps a system honest. It can blur precise terms, missing an exact product code, identifier, or name because semantic similarity smooths over the specific tokens that a keyword search would catch perfectly. It can also surface memories that are topically related but not actually useful, since closeness in meaning is not the same as relevance to the task. For these reasons, production systems rarely rely on vector search alone, instead combining it with keyword search and a reranking step. The full picture of how these methods complement each other is laid out in memory retrieval strategies.

A subtle but decisive factor is what text actually gets embedded in the first place. The same information stored as one long paragraph embeds very differently from the same content split into focused sentences, and retrieval quality depends heavily on this choice. Memories that are too long produce embeddings that average several ideas together and match any single query only weakly, while memories that are too short lose the surrounding context that gives them meaning. Tuning how information is chunked before it is embedded is one of the most effective and most overlooked levers for improving recall, frequently making a larger difference than swapping the embedding model or changing the similarity measure. Many recall problems that look like search failures are really chunking failures in disguise.

Vector Search in an Agent Memory System

Within a complete memory system, vector search is the engine of the retrieval stage. When the agent faces a new task, the system embeds the query, runs an approximate nearest neighbor search against the user's stored vectors, applies any metadata filters such as restricting results to the current user, and returns the top matches ranked by similarity. Those matches, possibly reranked, are then injected into the context window so the model can use them. Vector search is therefore not the whole memory system but the part that answers the central question of recall: given everything the agent has ever stored, which few memories matter right now.

Because vector search underpins recall, its configuration ripples through the whole agent. The embedding model sets the ceiling on how well meaning is captured, the similarity measure and index settings shape what gets returned, and the number of results retrieved trades recall against context budget. Tuning these is an ongoing part of operating an agent rather than a one-time setup, and it is the same machinery that powers retrieval augmented generation over any knowledge source. Get vector search right and the agent reliably finds what it knows; get it wrong and the agent appears forgetful even when the memory is sitting in the store.