Vector Databases for RAG: Comparison and Setup

What Vector Databases Do

A vector database stores high-dimensional numerical vectors and provides efficient nearest-neighbor search. When a RAG system needs to find documents relevant to a query, it converts the query into a vector and asks the database to find the stored vectors most similar to it. The database returns the closest matches along with their associated text and metadata, which are then passed to the language model as context.

Beyond basic similarity search, production vector databases provide metadata filtering (restricting search to vectors that match specific criteria like date range, source, or category), hybrid search (combining vector similarity with keyword matching), multi-tenancy (isolating data between different users or applications), and scaling mechanisms for growing collections.

Purpose-Built Vector Databases

Pinecone is a fully managed vector database service that requires no infrastructure management. Vectors are stored in namespaces within indices, with automatic scaling, replication, and backup. Pinecone's serverless offering charges based on storage and query volume rather than provisioned capacity, making it cost-effective for variable workloads. Key strengths include excellent hybrid search combining dense vectors with sparse keyword representations, strong metadata filtering, and enterprise features like encryption and access control. Pinecone is the default choice for teams that want to minimize operational overhead and are comfortable with a managed service.

Weaviate is an open-source vector database with a distinctive feature set including GraphQL-based querying, built-in vectorization modules that can embed text automatically during ingestion, and multi-modal support for text, images, and other data types. Weaviate can run self-hosted or as a managed service (Weaviate Cloud Services). Its vectorization modules mean you can send raw text to Weaviate and it handles embedding internally, simplifying the application code. The tradeoff is tighter coupling between the database and embedding model choice.

Qdrant is written in Rust and emphasizes performance and filtering capabilities. Its filtering system supports complex boolean queries on payload fields, enabling sophisticated retrieval constraints like "find documents similar to this query that were published after January 2025 and tagged as technical documentation." Qdrant supports both on-disk and in-memory storage modes, making it adaptable to different cost and latency requirements. Available as open source for self-hosting or as Qdrant Cloud.

Milvus targets large-scale deployments with billions of vectors. Built on a distributed architecture, Milvus separates compute, storage, and coordination into independent components that can scale horizontally. It supports GPU-accelerated indexing and search for maximum throughput. Milvus is the strongest option for teams with very large vector collections (100M+ vectors) that need high query throughput, but it comes with proportionally higher operational complexity.

Chroma focuses on simplicity and developer experience. It runs as an in-process library (no separate server needed) for development and small-scale production, with a client-server mode for larger deployments. Chroma's Python API is minimal and intuitive, making it the fastest option for prototyping RAG systems. It includes built-in embedding functions that handle vectorization automatically. The tradeoff is fewer enterprise features and less performance optimization for very large collections compared to Pinecone or Milvus.

Traditional Databases with Vector Extensions

PostgreSQL with pgvector adds vector similarity search to the world's most popular open-source relational database. For teams already running PostgreSQL, pgvector avoids introducing a new database service by keeping vectors alongside relational data in the same system. This simplifies operations, reduces infrastructure costs, and enables SQL queries that combine vector similarity with traditional relational filters in a single statement.

pgvector handles millions of vectors effectively with HNSW indexing and supports cosine, inner product, and L2 distance metrics. It does not match the performance of purpose-built vector databases at billion-vector scale, and it lacks some advanced features like automatic sharding and built-in hybrid search. But for teams with existing PostgreSQL infrastructure and moderate-scale RAG needs, it is a pragmatic choice that avoids operational complexity.

Index Types and Tradeoffs

Vector databases use approximate nearest-neighbor (ANN) algorithms to make search fast enough for real-time queries. The main index types each offer different tradeoffs between speed, accuracy, and memory usage.

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each vector connects to its nearest neighbors. Search navigates this graph from coarse to fine layers, finding approximate nearest neighbors very quickly. HNSW offers the best combination of speed and recall but requires storing the full graph in memory, which means memory usage scales linearly with collection size.

IVF (Inverted File Index) partitions vectors into clusters and searches only the closest clusters to the query. This reduces memory requirements compared to HNSW but may miss relevant vectors in adjacent clusters, producing lower recall. Increasing the number of clusters searched improves recall at the cost of latency.

Product Quantization (PQ) compresses vectors by dividing them into sub-vectors and replacing each with a codebook entry. This dramatically reduces storage requirements but introduces quantization error that reduces search accuracy. PQ is typically combined with IVF for large-scale deployments where memory constraints prevent storing full-resolution vectors.

Selection Criteria

Choose your vector database based on these practical considerations. For managed simplicity with strong production features, Pinecone or Weaviate Cloud. For self-hosted with maximum performance, Qdrant or Milvus. For development and prototyping, Chroma. For teams already running PostgreSQL who want to avoid a new service, pgvector. For billion-scale collections with high throughput requirements, Milvus.

Consider the total cost of ownership, not just the database pricing. Managed services charge for storage and queries but eliminate ops staffing. Self-hosted databases avoid per-query fees but require infrastructure, monitoring, and maintenance. For many teams, the ops cost of running a self-hosted vector database exceeds the API costs of a managed service, making managed options cheaper despite higher per-query pricing.

Hybrid Search Implementation

Hybrid search, combining vector similarity with keyword matching, has become a baseline requirement for production RAG in 2026. Different databases implement this differently. Pinecone supports sparse-dense hybrid search natively, allowing you to send both a dense vector and a sparse keyword representation in a single query. Weaviate uses BM25F for keyword search alongside vector search, with configurable fusion weights. Qdrant supports full-text search alongside vector search with payload-based filtering.

For databases that do not natively support hybrid search, you can implement it at the application level by running separate vector and keyword searches, then merging results using reciprocal rank fusion (RRF). RRF assigns each result a score based on its rank in each individual search and combines these scores to produce a final ranking. This approach works with any vector database and any keyword search system, giving you flexibility in your infrastructure choices.

Data Management and Operations

Production vector databases require ongoing operational attention. As documents are added, updated, and removed from the knowledge base, the vector index must be updated accordingly. Adding new vectors is straightforward in all databases. Updating existing vectors requires deleting the old version and inserting the new one, which means your application must track the mapping between source documents and their vector IDs. Deleting vectors when documents are removed prevents stale information from appearing in search results.

Backup and disaster recovery procedures should be established before going to production. Managed services handle this automatically, but self-hosted deployments need explicit backup schedules, tested restoration procedures, and monitoring for index corruption. The cost of re-embedding an entire document collection from scratch (if the index is lost) can be significant for large knowledge bases.

Scaling from Prototype to Production

Most RAG projects start with a small collection of a few thousand vectors during prototyping, then grow to hundreds of thousands or millions in production. This growth trajectory should inform your initial database choice. Starting with Chroma for a proof of concept is fine, but if you know the production knowledge base will contain 10 million documents, planning the migration to Qdrant or Pinecone early avoids a painful re-architecture later.

At small scale (under 100,000 vectors), nearly every vector database performs well and the choice barely matters. Differences in latency, recall, and throughput only become apparent at medium scale (100K to 10M vectors) and become critical at large scale (10M+ vectors). At medium scale, HNSW-based databases like Qdrant and Weaviate handle queries in single-digit milliseconds with high recall. At large scale, Milvus with GPU acceleration or Pinecone's serverless architecture provide the throughput and latency guarantees that self-hosted single-node deployments cannot match.

Memory consumption is often the binding constraint for self-hosted deployments. HNSW indexes store the full graph in RAM, consuming roughly 1.5 to 2 times the raw vector storage. A collection of 10 million vectors at 1536 dimensions requires approximately 57 GB for the raw vectors and 85 to 115 GB for the HNSW index. If your server has 128 GB of RAM, you are close to the ceiling with just 10 million vectors. Disk-based index modes (available in Qdrant, Milvus, and pgvector) trade some latency for dramatically lower memory usage, enabling much larger collections on modest hardware.

Cost Comparison in Practice

Vector database costs vary widely depending on scale, hosting model, and usage patterns. Pinecone serverless charges per million read units and per GB of storage, with no minimum commitment, making it cost-effective for variable workloads but potentially expensive at high sustained query volumes. A typical RAG application with 1 million vectors and 100,000 queries per month costs roughly $50 to $150 per month on Pinecone serverless.

Self-hosted databases eliminate per-query fees but introduce infrastructure costs. Running Qdrant or Weaviate on a cloud VM with 32 GB RAM and adequate storage costs $150 to $300 per month, but that same instance handles far more queries than you would pay for on a managed service. The breakeven point typically falls around 500,000 to 1 million queries per month, above which self-hosted becomes cheaper in pure infrastructure costs, though ops labor must be factored in.

pgvector is the cheapest option for teams already running PostgreSQL, since it requires no additional infrastructure. The only cost is the marginal storage and compute on your existing database, which for most moderate-scale RAG deployments adds minimal overhead to an already-provisioned PostgreSQL instance.

Multi-tenancy, the ability to isolate data between different users or applications, is a production requirement for SaaS deployments. Pinecone provides namespaces, Weaviate offers native multi-tenancy, and Qdrant supports collection-level isolation. Without built-in multi-tenancy, you must either run separate database instances per tenant (expensive and operationally complex) or implement tenant isolation in your application layer through metadata filtering (simpler but requires careful security review to prevent data leakage between tenants).