How to Deploy AI Agents with Docker in 10 Minutes

Install Docker and Verify Your Environment

Install Docker Desktop (macOS/Windows) or Docker Engine (Linux) from the official Docker website. Docker Desktop includes Docker Engine, Docker Compose, and a graphical management interface. Docker Engine on Linux is the lighter-weight option that provides only the command-line tools. Both support the same Compose file format and container operations.

Verify the installation by running docker --version and docker compose version in your terminal. Both commands should return version information without errors. If docker compose is not recognized, you may need to install the Compose plugin separately on older Linux installations. Docker Desktop bundles Compose automatically.

If you have an NVIDIA GPU and plan to run local models, install the NVIDIA Container Toolkit following the official NVIDIA documentation for your Linux distribution. After installation, restart the Docker daemon with systemctl restart docker. Verify GPU access by running the nvidia/cuda test container with the --gpus all flag and checking that nvidia-smi displays your GPU information inside the container.

Create a new directory for your agent project. This directory will contain your Compose file, Dockerfile, environment configuration, and agent source code. Keep your project directory organized from the start because the directory structure determines your Docker build context and affects which files are available during image builds.

Create Your Docker Compose File

Create a file named compose.yaml in your project directory. This file defines every service in your agent stack using YAML syntax. Start with three services: your agent runtime (built from a local Dockerfile), a model server (Ollama for local model inference or a placeholder if using cloud APIs), and a PostgreSQL database for persistent storage.

For the agent service, specify a build context pointing to your project directory and list the environment variables your agent needs, including the model server URL (http://ollama:11434 for local Ollama), the database connection string (postgres://agent:password@postgres:5432/agentdb), and any agent-specific settings. Add a depends_on section with condition: service_healthy for both the model server and database so your agent waits for its dependencies.

For the model server service, use the ollama/ollama image and configure GPU access through the deploy.resources.reservations.devices section. Add a named volume for model storage so downloaded models persist across container restarts. Include a health check that verifies the Ollama API is responding on port 11434.

For the database service, use the postgres:16 image with environment variables for the database name, user, and password. Add a named volume for data persistence and a health check using pg_isready. Declare all named volumes in the top-level volumes section at the bottom of your Compose file.

Configure Your Agent Runtime

Create a Dockerfile in your project directory that builds your agent container image. Start from python:3.11-slim as the base image, copy your requirements.txt file, install Python dependencies with pip, then copy your agent source code. Set PYTHONUNBUFFERED to 1 so logs appear immediately, and define an ENTRYPOINT that runs your agent script.

Create a .env file in your project directory for sensitive configuration values like API keys and database passwords. Reference this file in your Compose service definitions using the env_file key. Add .env to your .gitignore file immediately to prevent accidentally committing secrets to version control.

Configure your agent to connect to services using their Docker Compose service names as hostnames. Set MODEL_ENDPOINT to http://ollama:11434 for local Ollama, or to the cloud API URL if using a hosted model service. Set DATABASE_URL to the PostgreSQL connection string using postgres as the hostname (matching the service name in your Compose file).

If your agent requires a specific model to be available in Ollama, add an initialization step that pulls the model during first startup. You can use a startup script that runs ollama pull with your desired model name before starting the agent process. Alternatively, add an init container service in your Compose file that pulls the model and exits, with your agent depending on its completion.

Start the Stack and Verify Everything Works

Run docker compose up -d from your project directory to start all services in the background. Docker Compose builds your agent image (if using a build context), pulls any images not already cached locally, creates the named volumes, sets up the internal network, and starts the containers in dependency order.

Monitor the startup process with docker compose logs -f to see real-time output from all services. Watch for error messages, particularly database initialization logs (first run only), model download progress (if pulling a model), and agent connection attempts. The first startup takes longer because Docker needs to build your image, download base images, and potentially download model weights.

Verify that all services are running and healthy with docker compose ps. Every service should show a status of "Up" with "(healthy)" if you configured health checks. If any service shows "unhealthy" or has restarted, check its logs with docker compose logs followed by the service name to diagnose the issue.

Test your agent by sending a request to its API endpoint or connecting to its interface. If your agent exposes an HTTP API, send a test query and verify the response. Check that the response demonstrates the agent is using the model server (not returning static text) and that data persists in the database by sending a follow-up query that references the first interaction.

Once everything is working, stop the stack with docker compose down (preserves volumes and data) or keep it running. To restart later, run docker compose up -d again. Your data persists in the named volumes, so the agent picks up where it left off with its full conversation history and any cached models intact.