AI Agent Reliability Metrics Worth Tracking

Availability Metrics

Uptime percentage is the most fundamental reliability metric. It measures the fraction of time the agent system is operational and able to process tasks. Uptime is typically expressed in "nines": 99% (3.65 days of downtime per year), 99.9% (8.7 hours), 99.99% (52.6 minutes), 99.999% (5.3 minutes). Each additional nine requires roughly ten times the engineering effort to achieve.

For AI agents, availability must be measured at the task level, not just the process level. A running process that is stuck in a loop or producing garbage output is technically "up" but not functionally available. Define availability as the percentage of time the system can accept a new task and complete it successfully within expected time bounds.

Mean Time Between Failures (MTBF) measures the average time the system runs without a failure. A higher MTBF means fewer failures. Calculate it by dividing total operational time by the number of failures in that period. An MTBF of 72 hours means the agent crashes, on average, once every three days.

Mean Time To Recovery (MTTR) measures how quickly the system recovers after a failure. A lower MTTR means faster recovery. For systems with supervision trees and automatic restart, MTTR might be seconds. For systems requiring manual intervention, MTTR might be hours. The ratio of MTBF to MTTR determines practical availability: availability = MTBF / (MTBF + MTTR).

Mean Time To Detection (MTTD) measures how quickly failures are detected. This is often overlooked but critically important. An agent that has been silently producing incorrect results for two hours has a two-hour MTTD, and the damage during that window may be worse than a clean crash with instant detection. Invest in monitoring that reduces MTTD to seconds.

Error Rate Metrics

Task failure rate is the percentage of tasks that fail to complete successfully. Track this overall and broken down by task type, error category, and time period. A rising failure rate indicates a developing problem. A failure rate that spikes at specific times suggests a time-correlated dependency issue (like API rate limits during peak hours).

API error rate tracks failures in calls to external services, particularly LLM APIs. Separate error rates by provider, model, and error type. A 2% error rate from transient network issues is normal. A 15% error rate from rate limiting suggests you need to adjust your request patterns or upgrade your API tier.

Retry rate measures how often operations require retries before succeeding. A high retry rate with high eventual success means your retry strategy is working but your dependencies are unstable. A high retry rate with low eventual success means you are wasting resources on retries that will not help.

Circuit breaker trip rate tracks how often circuit breakers transition to open state. Each trip represents a dependency that failed badly enough to trigger protection. Track the duration of each open period and the dependency involved. Frequent trips on the same dependency indicate a systemic reliability problem with that service.

Recovery Metrics

Recovery success rate measures the percentage of automatic recoveries that succeed without human intervention. This is the most important metric for evaluating fault tolerance effectiveness. If automatic recovery succeeds 95% of the time, your system handles 19 out of 20 failures without anyone being paged. If it succeeds only 50% of the time, your fault tolerance has gaps that need attention.

Checkpoint restoration rate tracks how often checkpoint loading succeeds during recovery. Failed checkpoint restorations force cold starts, which lose more work. If checkpoint restoration fails frequently, investigate checkpoint corruption, version incompatibility, or storage reliability issues.

Data loss per incident measures how much completed work is lost when a failure occurs. With no checkpointing, data loss equals the entire task. With checkpointing every 30 seconds, maximum data loss is 30 seconds of work. Track this metric to evaluate whether your checkpoint frequency is adequate for your workload.

Escalation rate measures how often failures require human intervention after automatic recovery fails. This is the complement of recovery success rate. A declining escalation rate over time proves that your fault tolerance improvements are working. A rising escalation rate signals new failure modes that your automatic recovery does not handle.

Performance Under Failure

Degraded mode duration measures how long the system operates in a degraded state (using fallback models, reduced functionality, slower processing). Some degradation is expected and healthy, it means graceful degradation is working. But prolonged degradation indicates that recovery is incomplete or that the primary service is not returning to normal.

Task latency during recovery compares task processing time during normal operation versus during and immediately after a failure. Recovery often causes temporary performance degradation as caches are rebuilt, connections are re-established, and queued tasks are processed. Understanding this degradation helps set realistic expectations for post-failure performance.

Queue depth during outage tracks how many tasks accumulate while the agent is down or recovering. A growing queue during an outage is expected, but the recovery period must process the backlog without creating a secondary overload. If queue depth regularly causes post-recovery failures, you need either faster recovery or overflow handling.

Resource and Cost Metrics

Wasted API spend tracks API credits consumed by failed operations that had to be repeated. This directly quantifies the cost of failures and provides the financial case for fault tolerance investment. Compare wasted spend before and after implementing retry strategies and checkpointing to demonstrate ROI.

Incident response hours measures the engineering time spent responding to agent failures. Include diagnosis time, resolution time, and post-incident review time. This metric makes the human cost of poor reliability visible to management and supports headcount and tooling investment decisions.

Resource utilization during failure tracks CPU, memory, and network usage during failure and recovery. Restart storms, retry floods, and checkpoint loading can create resource spikes that affect other services. Understanding these patterns helps size infrastructure appropriately and implement resource limits.

Business Outcome Metrics

Task completion rate is the ultimate reliability metric. It measures what percentage of submitted tasks eventually complete successfully, regardless of how many retries, restarts, or fallbacks were needed along the way. A 99.5% task completion rate means that only 1 in 200 tasks fails permanently, even if the underlying system experiences frequent transient failures.

SLA compliance tracks whether the system meets its committed service level agreements. If you have committed to 99.9% availability and 95% of tasks completing within 60 seconds, this metric shows whether you are meeting those commitments. SLA metrics are the bridge between engineering reliability and business reliability.

User-perceived reliability measures reliability from the user perspective, which may differ from system-level metrics. An agent that crashes and recovers in 2 seconds may be perceived as perfectly reliable by users, while an agent that takes 30 seconds to respond (but never crashes) may be perceived as unreliable. Combine system metrics with user feedback to get the complete picture.

Setting Up a Reliability Dashboard

A single reliability dashboard should show the current state at a glance: overall availability, current error rates, circuit breaker states, active incidents, and trend lines for key metrics. This dashboard should be the first thing an on-call engineer checks during an incident and the primary artifact in weekly reliability reviews.

Organize the dashboard in layers. The top layer shows business metrics (task completion rate, SLA compliance). The middle layer shows system metrics (availability, error rates, recovery rates). The bottom layer shows infrastructure metrics (resource utilization, API costs). An incident typically starts with a business metric alert, which the engineer investigates by drilling into system metrics, then infrastructure metrics to find the root cause.

Set alert thresholds at two levels. Warning alerts fire when metrics cross thresholds that indicate developing problems (error rate exceeds 5%, MTTR exceeds 30 seconds). Critical alerts fire when metrics cross thresholds that indicate active impact (error rate exceeds 20%, availability drops below 99%). Warning alerts allow proactive response. Critical alerts demand immediate action.