Canary and Blue-Green Deployments for AI Agents

Blue-Green Deployments for Agents

A blue-green deployment maintains two identical production environments. The "blue" environment runs the current stable version. The "green" environment receives the new version. Once the green environment passes validation, traffic switches from blue to green in a single operation. Blue remains available for immediate rollback.

For AI agents, blue-green provides the cleanest version separation. There is no period where two versions serve traffic simultaneously, so you avoid the state compatibility challenges that arise when different versions process requests for the same user. Every user talks to either the old version or the new version, never a mix.

The traffic switch mechanism depends on your infrastructure. DNS-based switching changes a CNAME record to point from the blue load balancer to the green load balancer. This is simple but slow: DNS changes propagate over minutes, and cached DNS records can cause some users to hit the old environment for up to an hour after the switch. Load balancer-based switching changes the target group or backend service for an existing load balancer endpoint. This is fast (typically under a second) and avoids DNS propagation delays. Service mesh-based switching uses routing rules to redirect traffic at the application layer. Istio, Linkerd, and similar meshes support instant traffic switching with no infrastructure changes.

Before switching traffic, validate the green environment thoroughly. Run your behavioral test suite against the green environment using production-like inputs. Verify that all external integrations (LLM APIs, tool endpoints, databases) are connected and functional. Send a small number of synthetic requests and verify the responses manually. Only after this validation passes should you switch real traffic.

After the switch, monitor the green environment closely for at least 30 minutes. If any critical metric degrades, switch back to blue immediately. Keep the blue environment running for at least 24 hours after a successful switch. Only after the green environment has proven stable over a full business day should you decommission the blue environment (or repurpose it as the standby for the next deployment).

Canary Deployments for Agents

A canary deployment routes a small percentage of traffic to the new version while the majority continues to use the current version. The percentage increases gradually as confidence in the new version grows. If problems appear at any stage, only the canary percentage of users is affected, and traffic shifts back immediately.

Typical canary progression for AI agents: start at 5% of traffic, observe for 15-30 minutes. If metrics are healthy, increase to 15%, observe for 30 minutes. Continue to 30%, then 50%, then 100%, with observation periods at each step. The total rollout takes 2-4 hours. This is slower than a blue-green switch but provides much more observational data at each stage.

The traffic splitting mechanism is usually a load balancer weight or service mesh routing rule. AWS ALB weighted target groups, Kubernetes Ingress annotations, and Istio VirtualService weights all support percentage-based traffic splitting. The mechanism must support session affinity (sticky sessions) if your agent maintains conversational state. Without sticky sessions, a user might send their first message to the current version and their follow-up to the canary, which causes context loss and a broken experience.

Session affinity for canary deployments can be implemented through cookie-based routing (the load balancer sets a cookie that pins subsequent requests to the same backend), header-based routing (a consistent hash of the user ID determines which version handles the request), or explicit version assignment (a routing service assigns each new session to a version and stores the assignment). Cookie-based routing is simplest for HTTP-based agents. Header-based routing works better for agents accessed through APIs where cookies are not standard.

Evaluating Agent Quality During Rollout

The central challenge of canary deployments for AI agents is defining what "healthy" means. For a web application, healthy means low error rate and acceptable latency. For an AI agent, healthy also means good response quality, which is harder to measure automatically.

Layer your evaluation metrics from objective to subjective. Objective metrics that can be evaluated automatically in real time: error rate, latency percentiles, tool call success rate, token consumption per request, and conversation completion rate. These metrics should be compared between the canary and the stable version in real time. A dashboard showing side-by-side metrics for both versions lets operators see at a glance whether the canary is performing comparably.

Semi-automated quality metrics require more computation but can still run during the rollout: response relevance scores from a lightweight classifier or embedding similarity check, policy compliance scores from a rules engine that checks for prohibited content or format violations, and response consistency scores that verify the canary produces similar outputs to the stable version for equivalent inputs. These metrics take seconds to compute per request and can be aggregated over the observation window.

Subjective quality signals from users provide the ground truth but arrive slowly: thumbs up/down ratings, conversation abandonment patterns, follow-up question frequency (more follow-ups may indicate that the initial response was inadequate), and support escalation rates. These signals take hours or days to accumulate in statistically meaningful quantities, so they inform the decision to keep a deployed version rather than the initial rollout decision.

Define explicit promotion and rollback criteria before starting the canary. Example promotion criteria: error rate within 10% of the stable version, P95 latency within 20% of baseline, no increase in tool call failures, and automated quality score within 5% of baseline. Example rollback criteria: any 5xx error rate above 5%, P95 latency exceeding 3x baseline, or quality score dropping more than 15% below baseline. Having these criteria defined in advance prevents the ambiguity that causes teams to delay rollback decisions during an incident.

Shadow Deployments

Shadow deployments (also called dark launches) run the new version in parallel with the current version, processing the same inputs, but only delivering the current version's outputs to users. The new version's outputs are logged for comparison but never shown to users. This eliminates all user-facing risk during the evaluation period.

Shadow deployments are particularly valuable for AI agents because they let you compare response quality between versions on identical inputs. When you compare the current version's response to "How do I reset my password?" with the new version's response to the same question, you get a direct quality comparison that is impossible with separate canary traffic. You can evaluate whether the new version provides more accurate information, uses tools more effectively, or produces more concise responses.

Implementing shadow deployment for agents requires duplicating the request to both versions. The current version processes the request normally and returns the response to the user. The shadow version processes the same request, and its response is stored alongside the current version's response for later analysis. The duplication can happen at the load balancer level (mirroring requests), the application level (the routing service sends copies), or the queue level (both versions consume from the same message queue).

The primary cost of shadow deployment is doubling your LLM API usage. Every request generates two LLM calls: one for the live version and one for the shadow. For agents with high traffic volumes, this doubles the most expensive component of your infrastructure. Mitigate this by shadowing a percentage of traffic rather than all of it (10-25% is usually sufficient for meaningful comparison), or by running the shadow evaluation for a limited time window (2-4 hours during peak traffic).

Shadow deployments also have a state management consideration. If the new version writes to shared state stores (conversation history, memory, task queues), the shadow writes will contaminate production state. Run the shadow version against isolated state stores that mirror the production data but are not visible to the current version. This adds infrastructure complexity but prevents state corruption.

Implementation Tooling

The infrastructure you need depends on your deployment platform. On Kubernetes, Argo Rollouts is the most mature solution for automated canary deployments. Define a Rollout resource instead of a Deployment, specify your canary steps (percentages and pause durations), connect it to your metrics provider through an AnalysisTemplate, and Argo handles the traffic shifting, metric evaluation, and automatic promotion or rollback. Flagger offers similar functionality with a different configuration model and supports both Kubernetes Ingress and service mesh backends.

On AWS, CodeDeploy supports canary deployments for ECS and Lambda with built-in rollback on CloudWatch alarms. You define the traffic shifting configuration (percentage and interval), specify alarm conditions, and CodeDeploy handles the rest. For more sophisticated evaluation logic, combine CodeDeploy with Step Functions that run evaluation workflows during each canary stage.

For teams using service meshes, Istio VirtualService traffic splitting provides the most granular control. You define weight-based routing rules that split traffic between two service versions. Changing the weights is a simple configuration update, and the mesh handles the routing at the network level. Combine this with Prometheus metrics collection and Grafana dashboards for real-time side-by-side comparison of canary versus stable performance. LinkerdSplit in Linkerd provides equivalent functionality with a lighter operational footprint.

Without a dedicated canary tool, you can implement basic canary deployments using any load balancer that supports weighted target groups. Create two target groups (stable and canary), register the appropriate instances in each, and adjust the weights through the load balancer's API or console. This is less automated than dedicated canary tools but requires no additional infrastructure beyond what you already run.

Choosing the Right Strategy

Each strategy fits different situations, and most teams use more than one depending on the deployment type.

Use blue-green when the new version changes the state schema, when you need the fastest possible rollback, or when your infrastructure does not support percentage-based traffic splitting. Blue-green is also the right choice when the agent's response format changes in ways that break client applications, because gradual rollout would cause a percentage of clients to receive the new format before they can handle it.

Use canary when you want to observe real-world behavior incrementally, when the deployment is high-risk and you want to limit the blast radius, or when you need time to gather subjective quality signals before committing. Canary is the default strategy for most routine agent deployments because it provides the best balance of safety and speed.

Use shadow deployment when you are making significant changes to the agent's reasoning, prompt structure, or model, and you need to compare quality on identical inputs before exposing any users to the new version. Shadow deployments are the safest option for major version changes where even a 5% canary carries unacceptable risk.

For the highest-stakes deployments, combine strategies: run a shadow deployment first to validate quality, then execute a canary rollout to validate operational stability, and maintain the previous version in a blue-green standby for instant rollback. This layered approach provides maximum safety at the cost of a longer deployment timeline.