Deploying AI Agents on Kubernetes

Why Kubernetes for AI Agents

Kubernetes provides three capabilities that matter specifically for AI agent deployments: automated scaling, self-healing, and declarative configuration. Automated scaling lets you match agent capacity to demand without manual intervention. Self-healing restarts failed agent pods and replaces unhealthy instances automatically. Declarative configuration means your entire deployment, including replica counts, resource limits, health checks, and networking, is defined in version-controlled YAML files that can be reviewed, diffed, and rolled back like any other code artifact.

Beyond these fundamentals, Kubernetes has become the standard platform for ML and AI workloads in 2026. The ecosystem includes GPU schedulers, model serving frameworks, vector database operators, and observability tools that integrate natively with Kubernetes. If your organization already runs Kubernetes for other workloads, deploying agents alongside them reduces operational overhead and lets you share infrastructure, monitoring, and on-call practices across teams.

That said, Kubernetes is not always the right choice. For teams with no existing Kubernetes expertise, the operational complexity of running a cluster can exceed the complexity of the agent itself. Managed services like EKS, GKE, and AKS reduce this burden significantly, but they do not eliminate it. If your agent is a single process handling moderate traffic and you do not need autoscaling, a container running on a simple VM or managed service like Cloud Run, ECS Fargate, or a serverless platform may be a better fit.

Pod Design for Agent Workloads

The pod is the fundamental deployable unit in Kubernetes, and designing your agent pod correctly determines how well the rest of the Kubernetes machinery works with your workload.

Single-process pods vs. sidecar patterns. The simplest approach is one agent process per pod. This works well for agents that handle requests synchronously and do not require auxiliary processes. For more complex agents, the sidecar pattern adds supporting containers to the pod: a logging sidecar that ships agent traces to your observability platform, a proxy sidecar that manages outbound API calls with rate limiting and circuit breaking, or a cache sidecar that provides local access to frequently used embeddings or model weights. Sidecars share the pod's network namespace and can communicate with the main agent container over localhost, which eliminates network overhead for high-frequency internal communication.

Resource requests and limits. Set resource requests and limits explicitly for every agent container. Requests tell the scheduler how much CPU and memory the pod needs to function, and the scheduler uses these values to place pods on nodes with sufficient capacity. Limits cap the maximum resources a pod can consume, preventing a runaway agent from starving other workloads. For AI agents, memory limits are particularly important because agents that accumulate conversation context, tool results, and memory data can grow their memory footprint continuously over time. Set memory limits based on your worst-case context window size plus a 30% buffer, and implement application-level memory management that discards or summarizes old context before hitting the limit.

CPU requests for LLM-dependent agents. Most AI agents that call external LLM APIs spend the majority of their time waiting for API responses, not performing local computation. This means their CPU utilization is low even when they are busy. Set CPU requests based on actual utilization measurements, not on the number of concurrent requests the pod handles. A pod processing 50 concurrent requests might only need 0.5 CPU cores if each request spends 90% of its time waiting for API responses. Over-provisioning CPU wastes cluster resources and inflates costs.

Init containers for startup dependencies. Use init containers to handle setup tasks that must complete before the agent can start: downloading model weights, populating caches, waiting for dependent services to become available, or running database migrations. Init containers run sequentially before the main container starts, ensuring that the agent does not receive traffic until all dependencies are ready. This is cleaner than building startup logic into the agent itself, which can create complex state machines around initialization.

Health Checks That Work for Agents

Kubernetes health checks (probes) are designed for web applications that respond to HTTP requests in milliseconds. AI agents violate this assumption routinely because they process requests that take seconds or minutes to complete. Misconfigured probes are the number one cause of unnecessary pod restarts in agent deployments.

Startup probes should be configured with generous timeouts for agent pods. If your agent loads model weights, initializes vector database connections, or warms up caches during startup, these operations can take 30-120 seconds. Set the startup probe failure threshold high enough to accommodate the slowest reasonable startup (for example, failureThreshold: 30 with periodSeconds: 10 gives 5 minutes of startup time). Once the startup probe succeeds, Kubernetes switches to the liveness and readiness probes.

Liveness probes tell Kubernetes whether the pod is alive and should continue running. For agents, implement the liveness endpoint as a lightweight check that verifies the agent process is responsive, not that it can process a full request. A simple HTTP handler that returns 200 if the process is running and the event loop is not blocked is sufficient. Do not make the liveness probe call the LLM or execute tools. If the LLM provider is down, the agent is still alive; it just cannot process requests, which is a readiness problem, not a liveness problem.

Readiness probes tell Kubernetes whether the pod should receive traffic. This is where you check that the agent can actually do its job: the LLM API is reachable, required tools are available, the memory store is connected, and the agent has sufficient resources to handle new requests. When a pod fails its readiness probe, Kubernetes removes it from the service's endpoints so traffic routes to other healthy pods. Set readiness probe timeouts to at least double your LLM API's typical response time, because a slow API response should not cause the pod to be temporarily removed from service.

A common pattern for agents with variable request duration is to implement a concurrent request counter in the readiness probe. When the pod is processing the maximum number of concurrent requests it can handle efficiently, the readiness probe returns a failure, which stops new traffic from arriving until existing requests complete. This provides application-level load balancing that supplements Kubernetes service load balancing.

Autoscaling Agent Deployments

The Horizontal Pod Autoscaler (HPA) scales pod count based on metrics. For traditional web applications, CPU utilization is the standard scaling metric: when average CPU exceeds 70%, add pods; when it drops below 30%, remove them. For AI agents, CPU utilization is a poor scaling metric because agents spend most of their time waiting on external APIs, not consuming CPU. An agent pod at maximum capacity might show only 15% CPU utilization.

Better metrics for agent autoscaling include concurrent request count per pod, queue depth (if using a task queue), and response latency percentiles. These metrics reflect actual agent capacity utilization regardless of CPU usage. Kubernetes supports custom metrics through the Metrics API and external metrics through the External Metrics API. Use Prometheus with the Prometheus Adapter or a similar custom metrics solution to expose agent-specific metrics to the HPA.

Configure the HPA with conservative scaling behavior. Set the stabilization window to at least 5 minutes for scale-down to prevent thrashing when traffic fluctuates. Set scale-up behavior to add pods in small increments (1-2 at a time for small deployments, 10-20% at a time for larger ones) rather than jumping immediately to the calculated target. Aggressive scale-up can overwhelm dependent services (like the LLM provider's rate limit) and cause cascading failures.

For agents that require GPU resources, consider the Kubernetes cluster autoscaler alongside the HPA. When the HPA wants to add pods but no nodes have available GPU capacity, the cluster autoscaler provisions new GPU nodes. This chain of scaling (HPA scales pods, cluster autoscaler scales nodes) introduces additional latency because provisioning a new GPU node takes 3-10 minutes depending on the cloud provider. Account for this latency in your scaling strategy by maintaining a buffer of warm GPU capacity or by designing your agent to gracefully degrade to CPU-only operation when GPU resources are temporarily unavailable.

GPU Scheduling for Local Inference

Agents that run local model inference instead of calling external APIs require GPU resources in Kubernetes. GPU scheduling in Kubernetes works through the device plugin framework and resource requests. Each GPU-capable node registers its GPUs as extended resources, and pods request GPU allocations in their resource specifications.

GPU resources in Kubernetes are not divisible by default. A pod that requests one GPU gets exclusive access to an entire physical GPU, regardless of whether it uses 10% or 100% of the GPU's capacity. For agents that run small models that do not saturate a full GPU, this wastes expensive hardware. GPU sharing solutions like NVIDIA MPS (Multi-Process Service), MIG (Multi-Instance GPU) for A100 and H100 GPUs, and time-slicing through device plugins allow multiple pods to share a single GPU. MIG is the most robust option because it provides hardware-level isolation between workloads, preventing one pod's memory leak from affecting another.

When scheduling GPU workloads, use node affinity or taints and tolerations to ensure that GPU pods run on GPU-capable nodes and that non-GPU workloads do not get scheduled to GPU nodes. Tag GPU nodes with labels like accelerator=nvidia-a100 and use nodeSelector or affinity rules in your pod spec to match pods to appropriate hardware. This prevents the scheduler from placing your inference workload on a node with the wrong GPU type or from wasting GPU nodes on workloads that do not need them.

Managing State in Kubernetes Agents

Kubernetes is designed for stateless workloads. Pods can be killed, restarted, and rescheduled to different nodes at any time. AI agents, which maintain conversation history, memory, and task state, must reconcile this stateless paradigm with their inherently stateful nature.

The recommended pattern is to externalize all state to a service that persists independently of the agent pods. Redis, DynamoDB, PostgreSQL, or a dedicated memory store holds conversation histories, task state, and agent memory. The agent pod reads state from the external store when it begins processing a request and writes updated state back when it finishes. This means any pod can handle any request because the state is not tied to a specific pod instance. It also means pods can be killed and replaced without losing user context.

For agents that use local file storage (model weights, embedding caches, temporary files), use Kubernetes PersistentVolumes or node-local storage with appropriate lifecycle management. Model weights that are the same across all pods should be loaded from a shared read-only volume (like an EFS filesystem or a pre-populated PersistentVolumeClaim) rather than downloaded individually by each pod during startup. This eliminates redundant downloads and reduces startup time.

StatefulSets provide an alternative to Deployments when your agent needs stable network identifiers and persistent storage that follows the pod across rescheduling. However, StatefulSets complicate rolling updates and scaling because they maintain ordering guarantees that Deployments do not. Use StatefulSets only when your agent genuinely needs sticky identity, not as a shortcut for avoiding proper state externalization.

Networking and Service Mesh

Agent pods need outbound network access to LLM APIs, tool endpoints, and other external services. Configure Kubernetes network policies to allow outbound traffic to required destinations while restricting unnecessary access. This is especially important for agents that execute code or browse the web, where unrestricted network access creates security risks.

For agents that handle sensitive data, a service mesh like Istio or Linkerd provides mutual TLS between pods, traffic management with fine-grained routing rules, and observability features including request tracing and traffic metrics. The service mesh's traffic management capabilities are particularly useful for canary deployments, where you need to split traffic between the current and new agent versions by percentage. Istio's VirtualService resource lets you define traffic splitting rules declaratively, making canary rollouts a configuration change rather than an infrastructure change.

Configure network timeouts carefully for agent services. The default Kubernetes service timeout is often too short for agent requests that involve multi-step LLM chains. Set connection timeouts, request timeouts, and idle timeouts based on your agent's actual request duration distribution. A timeout set to 30 seconds will kill 10% of your requests if your P90 latency is 25 seconds, creating a terrible user experience for users whose requests happen to take slightly longer than average.