ChromaDB: The Open Source Embedding Database for AI Agents

Why Developers Start with ChromaDB

ChromaDB's appeal is speed to first result. Install it with pip install chromadb. Create a client. Create a collection. Add documents. Query. That entire workflow takes 60 seconds and produces working semantic search on your local machine. No account registration, no network calls, no configuration files.

The embedded mode runs as an in-process library, meaning ChromaDB runs inside your Python application with no separate server process. Data persists to a local directory on disk. This makes it ideal for Jupyter notebooks, local development, CLI tools, and any situation where adding infrastructure would slow down experimentation.

ChromaDB also handles embedding generation automatically. If you pass raw text strings instead of pre-computed vectors, ChromaDB uses a default embedding model (Sentence Transformers' all-MiniLM-L6-v2) to generate embeddings on the fly. You can swap in any embedding function, OpenAI, Cohere, or any Hugging Face model, through a simple interface. This auto-embedding feature eliminates the most common boilerplate in vector database setup.

Architecture and Storage

ChromaDB went through a significant architectural overhaul between v0.4 and v0.5. The current architecture (v0.5 and later) uses a Rust-based storage backend for improved performance and reliability, replacing the earlier pure-Python SQLite approach.

Data is organized into collections, each containing vectors, documents (the original text), metadata (key-value pairs), and unique IDs. Each collection can use a different embedding function and distance metric. The supported metrics are cosine similarity (default), L2 (Euclidean distance), and inner product (dot product).

ChromaDB supports two deployment modes. Embedded mode runs the database in your application's process, reading and writing to a local persistent directory. Client-server mode runs ChromaDB as a separate process accessible via HTTP, which allows multiple applications to share the same database and enables deployment patterns where the database runs on a different machine than the application.

The indexing layer uses HNSW for approximate nearest neighbor search, specifically the hnswlib library. Index parameters (M, ef_construction, ef_search) are configurable per collection, though the defaults work well for most development workloads. The default HNSW configuration uses M=16 and ef_construction=100, which provides good recall for datasets up to a few million vectors.

Core Operations

Adding data. The add() method accepts lists of IDs, documents (raw text), embeddings (pre-computed vectors), and metadata dictionaries. You can provide documents alone (ChromaDB generates embeddings), embeddings alone (you provide pre-computed vectors), or both (ChromaDB stores both and uses the embeddings for search). IDs must be unique within a collection. Adding a duplicate ID raises an error; use upsert() instead to update existing records.

Querying. The query() method accepts query texts (auto-embedded) or query embeddings (pre-computed) and returns the top-k nearest neighbors with their IDs, distances, documents, and metadata. The n_results parameter controls how many results to return, defaulting to 10.

Filtering. ChromaDB supports where clauses for metadata filtering and where_document clauses for text content filtering. Metadata filters use operators like $eq, $ne, $gt, $gte, $lt, $lte, $in, and $nin. Logical operators $and and $or combine multiple conditions. The where_document filter supports $contains for substring matching.

Filters are applied as post-processing after the vector search, which means that with very selective filters, you might receive fewer results than the requested n_results. This is a known behavior, not a bug. If you consistently get too few results, either increase n_results and trim the output, or restructure your data into separate collections instead of filtering within a single collection.

Updating and deleting. The update() method modifies existing records by ID. The upsert() method inserts new records or updates existing ones. The delete() method removes records by ID or by metadata filter. All operations are synchronous and take effect immediately.

Integration with LLM Frameworks

ChromaDB has first-class integrations with the major LLM and agent frameworks.

LangChain provides the Chroma vector store class that wraps ChromaDB collections with LangChain's standard interface. You can use it as a retriever in chains, as a memory backend for conversational agents, or as a document store for RAG pipelines. The integration supports persistent storage, custom embedding functions, and all of ChromaDB's filtering capabilities.

LlamaIndex offers a ChromaVectorStore class that integrates ChromaDB as an index backend. Documents ingested through LlamaIndex's document loaders and parsers are automatically embedded and stored in ChromaDB, and LlamaIndex's query engine handles retrieval and response generation.

Direct SDK usage is equally common. Many developers skip the framework layer entirely and use ChromaDB's Python client directly, especially for simpler applications or custom architectures that do not fit neatly into LangChain or LlamaIndex's abstractions.

Performance Characteristics

ChromaDB's performance profile is optimized for development and small-scale production, not for high-throughput or large-scale deployments.

Query latency at 100,000 vectors: 5 to 15ms (embedded mode), 8 to 25ms (client-server mode). These numbers are fast enough for interactive applications and AI agent workflows.

Query latency at 1 million vectors: 15 to 50ms (embedded mode), 25 to 80ms (client-server mode). Still acceptable for most use cases, but the gap with dedicated databases like Qdrant (2 to 5ms at 1 million) becomes noticeable in latency-sensitive applications.

Query latency at 5 million vectors: 50 to 200ms with significant variance depending on hardware and index configuration. At this scale, ChromaDB starts showing its limitations. Memory consumption grows substantially, cold-start times increase, and concurrent query performance degrades.

Write throughput: 2,000 to 5,000 inserts per second for pre-computed embeddings, 50 to 200 per second when ChromaDB generates embeddings (bottlenecked by the embedding model). Batch inserts using add() with lists are significantly faster than individual inserts.

Memory usage: ChromaDB's HNSW index keeps vectors in memory for fast search. At 1536 dimensions, each vector consumes approximately 6KB of RAM (plus HNSW graph overhead). One million vectors require approximately 8 to 10GB of RAM. This memory footprint limits the practical dataset size on machines with limited RAM.

When ChromaDB Is the Right Choice

Local development and prototyping. ChromaDB is unmatched for getting a vector search system running in minutes. Use it to validate your RAG pipeline architecture, test different embedding models, and iterate on chunking strategies before committing to production infrastructure.

Single-user AI agents. If your agent serves one user at a time, like a personal coding assistant, a local research tool, or a document Q&A system, ChromaDB provides all the vector search capability you need without any infrastructure complexity.

Educational and demo projects. Tutorials, workshops, and proof-of-concept demonstrations benefit from ChromaDB's zero-setup nature. Every participant can run the same code on their laptop without needing API keys or cloud accounts.

CI/CD testing. Integration tests that need a real vector database can spin up ChromaDB in-process with no Docker containers or network dependencies. This makes test suites faster, more reliable, and easier to run in any CI environment.

When to Migrate Away from ChromaDB

The signals that you have outgrown ChromaDB are clear. If your dataset exceeds 2 to 3 million vectors and query latency matters, migrate. If you need horizontal scaling across multiple nodes, migrate. If you need multi-tenant isolation with guaranteed performance per tenant, migrate. If you need production-grade monitoring, alerting, and observability, migrate. If concurrent query throughput exceeds what a single process can handle, migrate.

The migration path to production databases is straightforward because all major databases accept the same input format: vectors plus metadata. Export your ChromaDB data as vectors, metadata, and IDs, then import into Qdrant, Pinecone, or Weaviate using their respective batch import APIs. The only change in application code is swapping the database client, if you used LangChain or LlamaIndex's abstraction layer, it is a single configuration change.

The recommended production migration targets are: Pinecone if you want managed simplicity, Qdrant if you want self-hosted performance, and pgvector if you already run PostgreSQL and want to avoid adding new infrastructure.