How to Run AI Agent Evals in Production
Offline evaluation tests your agent against curated scenarios you anticipated. Production evaluation tests your agent against the actual distribution of user requests, including the scenarios you never imagined. These novel scenarios are where most production failures occur, because your test suite cannot cover inputs it never predicted. Production evaluation catches these failures within hours rather than waiting for user complaints that may take days or weeks to reach your team.
Design Your Sampling Strategy
Evaluating every production response is usually too expensive. A 10,000-request-per-day agent evaluated with LLM judges would cost $30-100 per day in judge API calls alone. Instead, sample strategically to maximize signal per dollar spent.
Uniform random sampling selects a fixed percentage of all responses regardless of content or context. This provides an unbiased estimate of overall quality. Start with 5-10% sampling for moderate-volume agents (1,000-10,000 requests per day) or 1-3% for high-volume agents (10,000+ per day). Even 1% sampling on a high-volume agent gives you 100+ scored examples per day, enough to detect meaningful quality shifts within 24 hours.
Stratified sampling over-samples high-risk categories and under-samples low-risk ones. Evaluate 20-50% of responses that involve tool calls (where formatting errors can cause hard failures), newly deployed capabilities (where untested edge cases lurk), long multi-step interactions (where compounding errors are most likely), and responses where the agent expressed uncertainty. Evaluate only 1-2% of simple, single-turn responses that match well-tested patterns. This concentrates evaluation budget on the cases most likely to reveal problems.
Confidence-based sampling evaluates responses where the agent's own confidence signals are low. If your agent produces a confidence score, internal uncertainty indicator, or takes an unusually high number of steps, flag that response for evaluation. Low-confidence responses fail at higher rates, so evaluating them preferentially catches more problems per evaluation dollar.
Trigger-based sampling evaluates all responses that match specific patterns: user follow-ups (suggesting the first response was inadequate), explicit negative feedback, agent retries or self-corrections, and responses that took unusually long to generate. These triggered evaluations complement random sampling by ensuring high-risk patterns are always scored regardless of sampling rate.
Build an Async Scoring Pipeline
Production evaluation must not add latency to user-facing responses. The scoring pipeline runs asynchronously: the agent responds to the user immediately, and the evaluation system scores the response afterward.
The pipeline architecture has four stages: capture, queue, score, and store. The capture stage intercepts agent responses (typically via a middleware or logging hook) and emits evaluation events containing the user input, agent output, full trace if available, and metadata (timestamp, user segment, task category). The queue stage buffers evaluation events in a message queue (SQS, Kafka, Redis Streams, or a simple database table) for processing at a controlled rate. The score stage pulls events from the queue, applies configured evaluators (deterministic checks, LLM judges, custom scorers), and produces evaluation results. The store stage writes results to a time-series database or analytics store for querying and visualization.
Design the scoring stage to be idempotent and resilient. If a scoring worker crashes mid-evaluation, the event should return to the queue for retry. If the judge API is temporarily unavailable, events should accumulate in the queue without loss. If scoring throughput falls behind event volume, the queue absorbs the burst and scoring catches up during lower-traffic periods.
For most teams, the simplest implementation is: log agent responses to a database table with a "scored" boolean flag, run a cron job every five minutes that selects unscored responses matching your sampling criteria, scores them with configured evaluators, and updates the records with scores. This requires no message queue infrastructure and handles moderate volumes (up to a few thousand responses per hour) easily. Scale to dedicated queue infrastructure only when volumes demand it.
Set Alerting Thresholds from Baseline Data
Before you can detect quality degradation, you need to establish what normal quality looks like. Run your production evaluation pipeline for two to four weeks on stable, known-good traffic to collect baseline data. Calculate the mean, standard deviation, and percentiles for each evaluation metric across this baseline period.
Set alert thresholds based on the baseline statistics. A common approach: alert when the rolling 24-hour average of any quality metric drops more than two standard deviations below the baseline mean. For metrics with natural variance (like LLM-judge scores), this prevents false alarms from normal fluctuation while catching genuine degradation. For metrics with low natural variance (like format compliance, which should be near 100%), set tighter thresholds because any drop is significant.
Configure tiered alerts: warning when a metric approaches the threshold (within one standard deviation), critical when it crosses the threshold, and emergency when multiple metrics degrade simultaneously or a single metric drops precipitously. Different tiers trigger different responses: warning alerts go to a monitoring channel for awareness, critical alerts page the on-call engineer, emergency alerts may trigger automatic rollback if you have that capability.
Recalibrate baselines after intentional changes. When you deploy a new model version, update a prompt, or modify tools, the quality distribution will shift (hopefully upward). After confirming the change is stable, update your baselines to reflect the new normal. Otherwise, you will get false alerts when quality fluctuates within the new (improved) range but crosses old (lower) thresholds.
Build Monitoring Dashboards
Dashboards transform raw evaluation data into actionable visibility. A production evaluation dashboard should answer three questions at a glance: is quality acceptable right now, is it trending up or down, and where should I investigate if something is wrong?
Overview panel: Current quality scores across key metrics (accuracy, faithfulness, format compliance, safety) with green/yellow/red indicators based on threshold comparison. Show the last 24 hours and the last 7 days to distinguish momentary dips from sustained trends.
Trend charts: Time-series plots of each metric at hourly or daily granularity. Overlay deployment markers (vertical lines showing when prompt changes, model updates, or tool changes were shipped) so you can correlate quality changes with specific deployments. Include confidence bands showing expected variance so genuine anomalies stand out visually.
Category breakdown: Quality scores segmented by task category, user segment, or capability. This reveals localized degradation that overall averages might mask. If the "product questions" category dropped 15% while "order tracking" stayed stable, the overall average might drop only 5%, but the category view immediately shows where the problem is.
Failure drilldown: A table of the lowest-scoring responses with links to full traces. When investigating an alert, engineers start here: read the worst cases, understand the failure pattern, and determine whether it is a systemic issue or isolated noise. Include the evaluator's reasoning (if using LLM judges) so the failure mode is immediately apparent without replaying the interaction.
Comparison view: Side-by-side metrics for the current version versus the previous version, or current day versus same day last week. This helps distinguish between version-specific regressions and seasonal/cyclical patterns in user behavior that affect quality scores.
Close the Feedback Loop
Production evaluation data should flow back into your development process, creating a continuous improvement cycle. Without this feedback loop, production evaluation is just monitoring; with it, production evaluation becomes your primary source of test cases and improvement targets.
Feed failures into your test suite. Every low-scoring production response is a candidate for your regression test suite. When the production evaluator flags a response as low quality, capture the input, the expected correct behavior (determined by human review of the specific case), and add it to your offline test dataset. Over time, this builds an evaluation suite that reflects your actual failure distribution rather than hypothetical scenarios.
Route borderline cases to human review. When the evaluator scores a response near the pass/fail threshold, it may not be confident enough to categorize it automatically. Route these ambiguous cases to a human review queue where a team member makes the final call. Their judgment both resolves the individual case and provides calibration data for improving your automated evaluator.
Track failure categories over time. Classify each low-scoring response by failure type (hallucination, tool error, refusal failure, formatting issue, incomplete answer). Track the distribution of failure types weekly. If "hallucination" was 10% of failures last month but is now 30%, that signals a specific problem to investigate rather than a general quality decline.
Prioritize improvements by impact. Use production evaluation data to prioritize which agent improvements will deliver the most user-facing value. If production evaluation shows that 40% of failures occur in one specific task category that represents 20% of traffic, fixing that category will improve overall quality more than improving a category that rarely fails.
Production evaluation is your last line of defense against shipping quality regressions to users. Start with simple random sampling at 5-10% scored by a single LLM judge, add stratified sampling and dashboards as volume grows, and always close the feedback loop by converting production failures into regression test cases.