How to Set Up a Vector Database for AI Agents

Step 1: Choose Your Database

Your choice depends on three factors: where you are in development, your hosting preference, and your scale expectations.

For local development and prototyping, use ChromaDB. Install with pip install chromadb and start embedding documents in four lines of Python. No API keys, no Docker, no configuration. When you are ready for production, migrate your data to a production database.

For production with minimal operational work, use Pinecone. Create a free account at pinecone.io, get your API key, and install the SDK with pip install pinecone. The free tier handles 100,000 vectors with enough query throughput for early production workloads.

For production with self-hosted control, use Qdrant. Run it with Docker: docker run -p 6333:6333 -v qdrant_data:/qdrant/storage qdrant/qdrant. You now have a production-ready vector database running locally. For remote deployment, run the same command on a VPS.

For adding vector search to an existing PostgreSQL database, use pgvector. Run CREATE EXTENSION vector; in your database and you are ready. No additional infrastructure required.

Step 2: Choose an Embedding Model

The embedding model determines the quality of your search results. A better model finds more relevant documents. The model also determines the vector dimension, which you must specify when creating your database collection.

For most projects: OpenAI's text-embedding-3-small (1536 dimensions, $0.02 per million tokens). It provides strong retrieval quality at low cost and is the most widely used model in production. Install the client with pip install openai.

For budget-conscious or high-volume projects: Open-source models like BGE-M3 (1024 dimensions) or Nomic Embed (768 dimensions) run locally without API costs. Install with pip install sentence-transformers and load the model. These run on CPU at 50 to 100 embeddings per second, or 500 to 1,000 per second on a GPU.

For search-optimized results: Cohere's embed-v4 (1024 dimensions) supports separate input types for documents and queries, which improves retrieval accuracy for search use cases. Install with pip install cohere.

Critical rule: use the same embedding model for all data in a collection and for all queries against it. Mixing models produces meaningless results because different models map concepts to different coordinates in vector space.

Step 3: Create a Collection

A collection (called an index in Pinecone, a class in Weaviate) is a named container for related vectors. Create one collection per distinct data type or use case, for example, one for your knowledge base and a separate one for conversation memory.

When creating a collection, you specify: the vector dimension (must match your embedding model's output, like 1536 for OpenAI's small model), the distance metric (use cosine for text embeddings, it is the right choice for almost all text use cases), and optionally, HNSW index parameters (the defaults work well for most workloads).

In ChromaDB, you create a collection with a single call to client.create_collection(). In Qdrant, you call client.create_collection() specifying the vectors configuration with size and distance parameters. In Pinecone, you create an index through the dashboard or API specifying dimension and metric. In pgvector, you create a table with a vector column: CREATE TABLE items (id bigserial PRIMARY KEY, content text, embedding vector(1536)).

For metadata, define the fields you will filter on during queries. Common metadata fields: source (where the document came from), category (topic classification), created_at (timestamp for freshness filtering), user_id (for multi-tenant isolation), and chunk_index (position within the original document for re-ordering results).

Step 4: Prepare Your Data

Raw documents must be split into chunks before embedding. Language models produce better embeddings for focused, coherent text segments than for entire long documents. The chunking strategy directly affects retrieval quality.

Chunk size: 200 to 500 words (roughly 250 to 650 tokens) is the practical sweet spot for most knowledge base and RAG applications. Smaller chunks are more precise but lose context. Larger chunks capture more context but dilute the embedding, making it harder to match specific queries.

Chunk overlap: Overlap adjacent chunks by 50 to 100 words to prevent information loss at chunk boundaries. Without overlap, a sentence that spans two chunks might not be retrievable by either chunk's embedding.

Chunk boundaries: Split on paragraph or section boundaries when possible, rather than at arbitrary word counts. A chunk that ends mid-sentence produces a weaker embedding than one that ends at a natural break point. LangChain's RecursiveCharacterTextSplitter and LlamaIndex's SentenceSplitter both handle this intelligently.

For each chunk, prepare the metadata you defined in step 3. Include the source document identifier, the chunk's position in the original document, and any categorical tags that will be useful for filtering during search.

Step 5: Generate Embeddings and Load Data

Pass each chunk through your embedding model to generate a vector. Then insert the vector, the original text, and the metadata into your collection.

Always use batch operations for loading data. Inserting 1,000 chunks in a single batch call is 5 to 10x faster than 1,000 individual inserts due to reduced network overhead and amortized index updates. Most databases and embedding APIs support batch sizes of 100 to 2,000 items per call.

For OpenAI embeddings, the API accepts batches of text and returns batches of vectors. Send chunks in batches of 100 to 500 (staying under the API's token limits per request). For local models via Sentence Transformers, the encode() method accepts a list of strings and returns a matrix of vectors.

Monitor your ingestion progress for large datasets. A 1 million chunk knowledge base takes approximately 30 to 60 minutes to embed with OpenAI's API (rate-limited to roughly 3,000 requests per minute on the standard tier) and 5 to 15 minutes to insert into the vector database. Log progress every 10,000 chunks so you can estimate completion time and catch errors early.

After loading, verify the data by checking the collection's vector count through the database's API or dashboard. The count should match the number of chunks you embedded.

Step 6: Run Your First Search Query

Test your setup by embedding a natural language question and searching the collection. The query workflow is: take the user's question, pass it through the same embedding model you used for your data, send the resulting vector to the database's search endpoint with a top-k parameter (start with 5 or 10), and examine the returned results.

Evaluate the results critically. Are the top results genuinely relevant to the query? Is the most relevant document in the top 3? Are irrelevant results appearing with high similarity scores? If the top results are not relevant, the issue is usually the embedding model (try a better one), the chunk size (try smaller or larger), or the data quality (garbage in, garbage out).

Test metadata filtering by adding a filter to your query. For example, search for similar documents but only from the "billing" category, or only from the last 30 days. Verify that the filter correctly narrows results without breaking relevance.

Adjust the ef_search parameter (for HNSW-based databases) if results seem incomplete. Higher values increase recall at the cost of latency. Start with the default, and increase to 100 or 200 if you suspect the index is missing relevant results.

Step 7: Connect to Your AI Agent

The final step integrates the vector database with your AI agent's workflow. The two most common integration patterns are RAG retrieval and conversational memory.

RAG retrieval: When the agent receives a user query, embed the query, search the vector database for the top 5 to 10 relevant chunks, format the retrieved text as context in the LLM prompt (typically as "Here is relevant information:" followed by the chunks), and let the LLM generate a response grounded in the retrieved context. LangChain's RetrievalQA chain and LlamaIndex's query engine automate this entire workflow.

Conversational memory: After each conversation turn, embed a summary of the exchange and store it in the vector database with metadata including the user ID, session ID, and timestamp. Before generating each response, search for relevant past conversations and include them in the prompt. This gives the agent long-term memory without filling the context window with entire conversation histories.

For both patterns, monitor the end-to-end latency. The total time is embedding the query (5 to 50ms for API, 10 to 100ms locally) plus database search (2 to 30ms depending on database and scale) plus LLM generation (500 to 3000ms). The vector database is rarely the bottleneck. If total latency is too high, optimize the LLM call first.