How to Scale Dockerized AI Agents

Step 1: Build an Optimized Agent Container Image

The container image should be as small as possible while containing everything the agent needs to run. Use a multi-stage build to separate the build environment from the runtime environment. The build stage installs dependencies and compiles any native extensions. The runtime stage copies only the necessary artifacts into a minimal base image (python:3.12-slim, node:20-slim, or similar).

Externalize all configuration using environment variables or a configuration service. The image should not contain API keys, model names, endpoint URLs, or any other values that change between environments. This allows the same image to run in development, staging, and production with different configurations, eliminating the "works on my machine" problem.

All state must live outside the container. Conversation histories go in Redis, task state goes in the database, cached data goes in a shared cache. Nothing that would be lost when the container stops should exist inside the container. This is the prerequisite for horizontal scaling, because any container instance must be able to process any task without access to data from other instances.

Configure logging to write to stdout/stderr in structured JSON format. Container orchestrators capture stdout/stderr automatically and route it to centralized logging services. File-based logging inside containers creates volume management problems and makes log aggregation difficult.

Step 2: Configure Resource Limits and Requests

Resource limits prevent a misbehaving container from consuming all host resources and affecting other containers. Resource requests guarantee that the container has the minimum resources it needs to function correctly. Both are critical for stable scaling.

For AI agent workers, CPU requests should match the baseline processing load (typically 0.5-1 CPU for a Python-based agent), and CPU limits should allow bursts for prompt assembly and response parsing (typically 2-4 CPU). Memory requests should cover the agent runtime, loaded libraries, and typical working data (typically 512MB-1GB). Memory limits should include headroom for processing large requests (typically 2-4GB).

Setting limits too low causes container restarts (OOM kills) under normal operation. Setting them too high wastes cluster resources and reduces the number of containers that can run on each node. Profile your agent under realistic workloads to find the right values. Monitor actual resource usage after deployment and adjust based on observed patterns rather than initial guesses.

Step 3: Implement Health Checks and Readiness Probes

Health checks and readiness probes serve different purposes. A health check (liveness probe in Kubernetes) determines whether the container is functioning. If the health check fails, the orchestrator restarts the container. A readiness probe determines whether the container is ready to accept new work. If the readiness probe fails, the orchestrator stops routing new tasks to the container but does not restart it.

For AI agents, the health check should verify that the agent process is running and responsive. A simple HTTP endpoint that returns 200 if the process is alive is sufficient. The readiness probe should be more sophisticated: verify that the agent can connect to Redis, can reach the LLM API endpoint (a lightweight connectivity check, not a full inference call), and is not in a shutdown/draining state.

Set the health check interval to 10-30 seconds and the failure threshold to 3 consecutive failures before restart. This prevents unnecessary restarts from momentary hiccups while still catching genuinely failed containers within a minute.

Step 4: Set Up Queue-Based Auto-Scaling

Standard CPU-based auto-scaling does not work well for AI agent containers because workers spend most of their time waiting for LLM API responses, keeping CPU utilization low even when the system is at capacity. Queue depth is the correct scaling signal.

In Kubernetes, use the Horizontal Pod Autoscaler (HPA) with a custom metric from your queue system. The metric should be queue depth divided by the number of active worker pods. When this ratio exceeds your target (for example, more than 8 pending tasks per worker), HPA adds pods. When it drops below a lower threshold (for example, fewer than 2 pending tasks per worker), HPA removes pods.

For non-Kubernetes environments, implement the same logic in a simple scaling script. AWS ECS supports custom metric scaling through CloudWatch. Docker Compose deployments can use a cron job that queries queue depth and adjusts the replica count via the Docker API. The algorithm is the same regardless of the orchestration platform: measure queue depth per worker, scale up when above threshold, scale down when below.

Configure minimum and maximum replica counts. The minimum should be at least 2 (for redundancy even during off-peak hours). The maximum should be set based on your LLM API rate limit, because adding workers beyond what the rate limit can support provides no throughput benefit while increasing API contention.

Step 5: Deploy with Rolling Updates

Rolling updates replace old container versions with new ones gradually, maintaining system availability throughout the deployment. The orchestrator starts new containers running the updated version, waits for them to pass readiness probes, then terminates old containers one at a time.

For AI agents, the termination process must account for long-running requests. When a container receives a shutdown signal (SIGTERM), it should stop accepting new tasks from the queue, wait for in-progress tasks to complete (up to a configurable timeout, typically 60-120 seconds), and then exit cleanly. This is called graceful shutdown or task draining.

Configure the orchestrator termination grace period to match your maximum expected task duration. If your longest task takes 90 seconds, set the grace period to at least 120 seconds. This gives in-progress tasks time to complete naturally rather than being forcefully terminated, which would result in incomplete results and tasks that need to be reprocessed.