What Is a Vector Database

The Core Concept: Storing Meaning as Numbers

Every piece of text, image, or audio can be converted into a list of numbers called an embedding. An embedding model, typically a neural network trained on billions of examples, reads the input and outputs a fixed-length array of floating-point numbers. For text, a common embedding might have 1536 dimensions (OpenAI's text-embedding-3-small) or 1024 dimensions (Cohere's embed-v4). Each number in the array captures some aspect of the input's meaning, though individual dimensions do not map to human-readable concepts.

When two pieces of content are semantically similar, their embeddings are close together in this high-dimensional space. When they are unrelated, their embeddings are far apart. A vector database exploits this property by storing millions or billions of these embeddings and finding the closest ones to any given query embedding, fast.

The distance between two vectors is measured using mathematical functions. Cosine similarity is the most common for text, measuring the angle between two vectors on a scale from -1 (opposite meaning) to 1 (identical meaning). A cosine similarity of 0.92 between two document embeddings means they are very closely related. A score of 0.3 means they share only loose thematic overlap.

How a Vector Database Differs from a Traditional Database

In a relational database like PostgreSQL or MySQL, you query with exact conditions. SELECT * FROM products WHERE category = 'electronics' AND price < 500. The database scans an index, finds exact matches, and returns them. There is no notion of "close enough" or "similar to." Either a row matches the condition or it does not.

In a vector database, you query with a vector and ask for the nearest neighbors. "Find the 10 stored vectors that are closest to this query vector." The database uses approximate nearest neighbor algorithms to find them without scanning every stored vector. Results come back ranked by similarity, not filtered by equality.

This fundamental difference makes vector databases essential for any application that deals with unstructured data, text, images, audio, video, code, where meaning cannot be captured by structured fields and exact filters alone.

It is worth noting that the line between traditional and vector databases is blurring. PostgreSQL added vector support through the pgvector extension. Redis has vector search capabilities. Elasticsearch supports dense vector fields. But purpose-built vector databases like Qdrant, Pinecone, and Weaviate still outperform these hybrid approaches at scale because their entire storage engine, indexing, and query execution are optimized specifically for vector operations.

Why AI Agents and LLMs Need Vector Databases

Large language models have two critical limitations that vector databases solve. First, LLMs have a finite context window. Even the largest models in 2026 cap at 128K to 200K tokens, which is roughly 300 pages of text. An AI agent that needs to reference a company's entire documentation, thousands of support tickets, or years of customer interactions cannot fit all that data into a single prompt.

Second, LLMs have static training data. They know what was in their training set, but they do not know about your company's internal processes, your latest product updates, or what a specific customer said last Tuesday. Vector databases provide the mechanism for feeding this real-time, private, domain-specific data to the LLM at query time.

The standard pattern is retrieval-augmented generation (RAG). Your documents get embedded and stored in the vector database once. When a user asks a question, the agent embeds the question, searches the vector database for the most relevant documents, pulls the top results, and injects them into the LLM prompt as context. The LLM generates its response based on this retrieved context rather than relying solely on its training data.

Beyond RAG, vector databases enable persistent memory for agents. Conversation history, user preferences, learned facts, and past decisions get stored as embeddings. When the agent processes a new interaction, it retrieves relevant memories and uses them to inform its response. This is how an AI agent "remembers" that you prefer Python over JavaScript, or that your team decided to use Kubernetes three months ago.

The Key Components of a Vector Database

Storage Engine. The layer that persists vectors and metadata to disk or memory. Some databases use memory-mapped files (Qdrant), some use custom columnar formats (LanceDB), and some use existing storage engines like SQLite (ChromaDB) or RocksDB (Weaviate).

Index. The data structure that enables fast approximate nearest neighbor search. HNSW (Hierarchical Navigable Small World) is the dominant index type, used by Qdrant, Weaviate, pgvector, and ChromaDB. It builds a multi-layer graph of connections between vectors, allowing queries to navigate to the nearest neighbors through a series of hops rather than scanning every vector.

Query Engine. The component that processes incoming search requests, traverses the index, applies metadata filters, computes distance scores, and returns ranked results. The query engine determines whether filters are applied before or after vector search, which significantly affects both performance and result quality.

Metadata Store. Every vector is stored alongside metadata, structured fields like document titles, timestamps, categories, user IDs, or any other key-value pairs. Metadata enables filtered search: "find the most similar vectors, but only from documents tagged as 'engineering' and created in the last 30 days."

API Layer. The interface through which applications interact with the database. Most vector databases provide REST APIs and client libraries for Python, JavaScript, Go, and other languages. Weaviate uses GraphQL. Pinecone, Qdrant, and ChromaDB use REST with gRPC options for high-throughput scenarios.

When You Should and Should Not Use One

Use a vector database when your application needs to find semantically similar content across a large corpus, when you are building RAG systems for LLMs, when you need persistent memory for AI agents, when you want to implement recommendation systems based on content similarity, or when you need to deduplicate or cluster unstructured data.

Do not use a vector database when your data is purely structured and your queries are exact filters (use PostgreSQL or MySQL). Do not use one when your dataset is small enough to fit in the LLM's context window, a few hundred documents. Do not use one when keyword search is genuinely sufficient for your use case, if users always search with precise product SKUs or order numbers, vector search adds complexity without benefit.

The most common mistake is reaching for a dedicated vector database too early. If you have fewer than 100,000 vectors and your application already runs PostgreSQL, adding the pgvector extension is almost always the right first step. You can migrate to a dedicated vector database later when scale or performance requirements demand it.