Zero-Downtime AI Agent Updates
The goal is not just uptime. A deployment that keeps the server online but causes users to lose their conversation context, restart mid-task, or receive inconsistent responses is not truly zero-downtime from the user's perspective. True zero-downtime updates are invisible to every user, including those with active sessions during the deployment.
Prepare the New Version
Build the new version's deployment artifact (container image, function package, or binary) and deploy it to your staging environment. Run the full test suite including behavioral evaluations. Verify that the new version can read state created by the current version. This backward compatibility check is the most critical preparation step, because it determines whether active sessions can transition between versions without errors.
Test the state compatibility explicitly: export a sample of current production state (conversation histories, memory records, task progress), load it into the new version, and verify that the new version processes it correctly. If the new version introduces state schema changes, verify that the migration logic handles every variant of the current state format, including edge cases like empty fields, maximum-length conversations, and partially completed tasks.
Pre-pull the new version's container image to all nodes in your cluster. When the rolling update begins, pods that already have the image cached start in seconds instead of minutes. On Kubernetes, use a DaemonSet or a scheduled job to pull the image to every node before initiating the rollout. This eliminates the most common source of update latency and prevents a situation where one slow image pull delays the entire deployment.
Stop New Session Assignment to Old Instances
Begin the transition by routing all new sessions (new conversations, new task requests) to the new version while existing sessions continue on the current version. This creates a natural drain: as existing sessions end organically, the old version's load decreases until it reaches zero.
Implement this through your load balancer's routing rules. Mark old instances as "draining" in the load balancer, which stops new connections but allows existing connections to continue. On Kubernetes, this happens automatically when you update a Deployment: new pods with the updated version start receiving traffic while old pods are marked for termination. The terminationGracePeriodSeconds setting controls how long old pods have to complete their active work before being killed.
For agents with WebSocket or long-lived HTTP connections, the load balancer drain must be connection-aware. Some load balancers (ALB, HAProxy) support connection draining natively, allowing existing WebSocket connections to continue while routing new connections to new backends. If your load balancer does not support connection-level draining, implement it at the application level: the agent process stops accepting new connections when it receives a SIGTERM signal but continues processing requests on existing connections.
Drain Active Connections Gracefully
Set a drain timeout that gives active requests enough time to complete naturally. For agents that process short request-response interactions, a 30-60 second drain timeout is usually sufficient. For agents that handle long-running tasks or multi-step workflows, the timeout may need to be 5-15 minutes.
Monitor the drain progress. Track how many active requests remain on old instances and how quickly they are completing. If the drain is proceeding normally (active request count decreasing steadily), let it complete. If the count stalls (requests are not completing), investigate whether a specific request is stuck and handle it individually rather than extending the timeout for all requests.
Handle the timeout expiration gracefully. When the drain timeout expires and some requests are still active, you have two options. First, force-terminate the requests and accept that those specific users will experience a disruption. This is the simpler approach and is appropriate when the stuck requests are edge cases that should not block the entire deployment. Second, extend the timeout for the specific instances with remaining active requests while allowing the rest of the deployment to proceed. This preserves zero-downtime for the stuck users at the cost of running old instances longer.
Implement a graceful shutdown handler in your agent process. When the process receives a termination signal (SIGTERM on Linux), it should stop accepting new requests, wait for active requests to complete (up to the drain timeout), flush any buffered data (logs, metrics, state updates), close connections to external services cleanly, and then exit. The graceful shutdown handler prevents data loss and connection errors that occur when a process is killed abruptly.
Migrate Persistent State
If the new version changes the state schema, persistent state must be migrated so the new version can read it. The migration timing depends on the scope of the change.
Additive changes (new fields with defaults) require no migration. The new version reads old state, finds the new fields missing, and uses defaults. The old version reads new state, finds extra fields it does not recognize, and ignores them. Both versions coexist without issues.
Format changes (restructured fields, renamed keys, new encoding) require explicit migration. The safest approach is a two-phase migration. Phase 1: the new version reads both old and new formats, writing in the new format. During the transition period, both versions can process any state record. Phase 2: after the old version is fully decommissioned, run a batch migration to convert remaining old-format records to the new format. This two-phase approach eliminates the risk of incompatible state during the transition.
For conversation state specifically, do not attempt to migrate mid-conversation. Let active conversations complete on the version that started them. New conversations start on the new version with the new state format. This avoids the complexity and risk of transforming state for an active session and is the most common approach in production agent deployments.
Decommission Old Instances
After all active sessions on the old version have completed and all traffic is flowing to the new version, shut down the old instances. Do not rush this step. Keep the old instances available but idle for at least 30 minutes after the last active session ends, in case a delayed request or retry arrives. After the idle period, terminate the old instances.
Retain the old version's deployment artifact (container image, configuration, state backup) for at least one week after decommissioning. If a problem is discovered days later that traces back to the deployment, you need the old version available for comparison testing and potential rollback.
Verify the decommissioning is complete. Check that no old pods, containers, or processes are still running. Confirm that the old version's health check endpoints are no longer responding. Verify that the load balancer's target group contains only new-version instances. Clean up any temporary resources created during the transition: shadow state stores, migration scripts, and pre-pull jobs.
Special Cases for Agent Updates
Prompt-only updates are the simplest zero-downtime update for agents. If prompts are loaded from a configuration store rather than compiled into the binary, update the prompt in the store and let agents pick up the new prompt on their next request or at the next configuration refresh interval. No container replacement is needed, no connections drain, and no state migration occurs. This is one of the strongest arguments for loading prompts from external configuration rather than bundling them in the deployment artifact.
Model version updates without code changes can also be zero-downtime if the model endpoint URL is configurable. Update the model endpoint in configuration, and agents route their next LLM call to the new model version. For agents that cache the model client, implement a configuration watcher that reinitializes the client when the model endpoint changes. This avoids a full deployment for a change that only affects which model the agent calls.
Agents with persistent WebSocket connections need special handling because the connection itself carries state. When the old instance shuts down, the WebSocket connection drops, and the client must reconnect. Design your client to handle reconnection automatically: detect the disconnection, reconnect to the load balancer (which routes to a new-version instance), and resume the conversation from the last acknowledged message. Store enough state server-side that the new instance can continue the conversation without asking the user to repeat themselves.
Measuring Zero-Downtime Success
After every deployment, verify that zero-downtime was actually achieved. Check these metrics:
Error rate during the deployment window. Compare the error rate during the deployment to the baseline error rate for the same time period on a non-deployment day. Any increase above normal variance indicates that some users experienced disruption.
Connection drops during the deployment window. Monitor WebSocket disconnections, HTTP connection resets, and TCP errors. A spike during the deployment indicates that the drain or shutdown process is not handling active connections correctly.
Response latency during the deployment window. A temporary increase in latency is expected during rolling updates as the system operates at reduced capacity. If the latency increase is too large (more than 50% above baseline) or lasts too long (more than 10 minutes), the rolling update configuration needs adjustment: increase the number of instances available during the transition, or slow down the rollout speed.
Conversation continuity. Check whether any users had to restart their conversations during the deployment. This is the most important metric for agent deployments because it directly measures the user-visible impact. Track conversation session IDs and verify that no sessions were interrupted during the deployment window.
Zero-downtime updates for AI agents require more than just keeping the server online. They require preserving every active conversation, completing every in-flight task, and ensuring that no user notices the version change. Externalize state, implement graceful shutdown handlers, configure drain timeouts generously, and measure success by conversation continuity, not just uptime.