How to Build a Fault-Tolerant Agent System

Design the Process Architecture

Start by drawing the process boundaries. Each agent should run in its own isolated process (OS process, container, or lightweight virtual process depending on your platform). Isolation means that a crash in one agent cannot corrupt memory or state in another agent. This is the single most important design decision for fault tolerance.

Define a supervision hierarchy. Every worker process needs a supervisor that detects crashes and restarts the worker. Group workers under supervisors by failure domain: agents that share the same external dependencies should be managed by the same supervisor, so the supervisor can implement coordinated recovery (like switching all agents to a fallback model when the primary model goes down).

Choose your restart strategy for each supervisor. Use one-for-one when agents are independent (most common). Use one-for-all when agents share state that becomes inconsistent after a partial failure. Set restart intensity limits (maximum 5 restarts per 60 seconds is a good starting point) to prevent restart storms.

Design the communication model between agents. Message passing (rather than shared memory) preserves isolation. Use message queues for asynchronous communication and request-response patterns for synchronous calls. Every inter-agent call should have an explicit timeout to prevent one hung agent from blocking others.

Implement Retry and Circuit Breakers

Wrap every external call (model API, tool execution, database query, HTTP request) in a retry wrapper with exponential backoff and jitter. Configure retry parameters based on the specific service: 3 retries with 1-second base for fast APIs, 5 retries with 2-second base for slower services. Always set a maximum retry count and a maximum total timeout.

Add a circuit breaker for each external dependency. Configure the failure threshold (50% error rate over 20 requests is a good default), the open timeout (30 seconds for fast-recovering services, 2 minutes for slower ones), and the half-open probe count (3 to 5 successful requests before fully closing).

Wire the circuit breaker and retry logic together. Retries happen inside the circuit breaker: when the breaker is closed, failed requests are retried according to the retry policy. When the breaker is open, requests fail immediately without attempting any retries. This prevents retries from prolonging an outage.

Classify errors to determine retry behavior. Network timeouts, HTTP 429 (rate limited), and HTTP 503 (service unavailable) are retryable. HTTP 400 (bad request), 401 (unauthorized), and 404 (not found) are not retryable. Model-specific errors like content filter rejections are usually not retryable with the same input.

Add State Checkpointing

Implement state checkpointing that saves agent progress after each significant step. Define what constitutes a "step" for your workload: it might be after each tool call, after each model response, or after each completed subtask.

Choose a checkpoint storage backend appropriate for your checkpoint frequency and size. Redis for high-frequency small checkpoints (every few seconds, under 1 MB). PostgreSQL or SQLite for moderate-frequency larger checkpoints (every few minutes, any size). S3 for infrequent large checkpoints (every 10+ minutes, any size).

Include a version number and timestamp in every checkpoint. The version number allows the recovery process to handle checkpoints from older code versions. The timestamp allows cleanup policies to expire old checkpoints.

For conversation history, implement summarization checkpoints that compress long histories into compact summaries. This prevents checkpoints from growing unboundedly as the agent processes more steps. On recovery, the agent loads the summary and continues with reduced but functional context.

Test checkpoint restoration regularly. A checkpoint that cannot be loaded is worse than no checkpoint because it gives false confidence. Include checkpoint restoration in your automated test suite.

Build Graceful Degradation

Define a model fallback chain for every model dependency. When the primary model is unavailable, the agent should automatically switch to the next model in the chain. Test each fallback to verify it produces acceptable results for your use case.

For each tool and capability, define what happens when it is unavailable. Can the agent continue without it? Can it use an alternative? Should it queue the task for later? Document these degradation paths explicitly so that the implementation matches the design.

Implement capability-aware routing that knows the current health status of all dependencies and adjusts agent behavior accordingly. When the vector database is down, skip RAG retrieval and use the model knowledge directly. When the web scraper is down, use cached data or inform the user that live web data is temporarily unavailable.

Set Up Monitoring and Alerting

Instrument every key operation with metrics: model API calls, tool executions, task completions, errors, and resource usage. Use a metrics library that supports dimensional tags for filtering by agent, task type, and error category.

Configure health check endpoints (liveness and readiness) for each agent process. Wire these into your orchestration platform (Kubernetes, Docker, systemd) so that crashed or hung processes are detected and restarted automatically.

Set up structured logging (JSON format) with consistent fields across all components. Send logs to a centralized logging system (Elasticsearch, Loki, CloudWatch) where they can be searched and correlated across agents.

Define alerting rules at two levels: warning alerts for developing problems (error rate over 5%, latency over 10 seconds) and critical alerts for active impact (error rate over 25%, agent unresponsive). Route critical alerts to the on-call engineer.

Build a reliability dashboard that shows business metrics at the top (task completion rate, SLA compliance), system metrics in the middle (error rates, circuit breaker states), and infrastructure metrics at the bottom (CPU, memory, API costs).

Test Failure Scenarios

Fault tolerance that has never been tested is fault tolerance that probably does not work. Deliberately inject failures to verify every recovery mechanism.

Kill agent processes at random intervals and verify that supervisors restart them correctly. Block network access to model APIs and verify that circuit breakers trip and fallback models activate. Corrupt checkpoint files and verify that the agent falls back to cold start gracefully.

Test under load. Recovery that works for a single agent might fail when 100 agents try to recover simultaneously, overwhelming the checkpoint storage, the fallback model, or the monitoring system. Scale your failure tests to match your production scale.

Schedule regular failure drills (weekly or monthly) where the team practices responding to realistic failure scenarios. This builds operational muscle memory and reveals gaps in documentation, runbooks, and tooling. The goal is to make failure response routine rather than panicked.