How to Use LLM as a Judge for Agent Testing
LLM-as-judge evaluation solves the fundamental problem that most agent outputs cannot be scored with simple string matching or rule-based checks. When an agent writes an email, summarizes a document, answers a research question, or generates a plan, there are multiple valid outputs and quality exists on a spectrum. Human evaluation is the gold standard but does not scale. LLM judges provide a middle path: automated scoring that handles nuance and subjectivity, running at the speed and cost of an API call.
Define Your Evaluation Criteria as a Rubric
A rubric is a scoring guide that tells the judge model exactly what to look for and how to assign scores. Without a clear rubric, the judge produces inconsistent scores because it applies its own implicit criteria, which drift between calls and between model versions.
Write your rubric with explicit anchors for each score level. For a 1-5 scale, describe what a 1 looks like (completely wrong, irrelevant, harmful), what a 3 looks like (partially correct, addresses the question but misses key points), and what a 5 looks like (comprehensive, accurate, well-structured, directly addresses all aspects of the query). Include concrete examples for each level when possible.
Separate your rubric into independent dimensions rather than trying to score everything on one scale. Common dimensions include: factual accuracy (are the claims correct), completeness (does it cover all relevant aspects), relevance (does it answer what was asked), clarity (is it well-written and understandable), and safety (does it avoid harmful content). Scoring each dimension independently produces more actionable feedback than a single holistic score because you can identify exactly which quality aspects need improvement.
Keep rubric descriptions concrete and behavioral rather than abstract. Instead of "the answer should be high quality," write "the answer should contain at least three specific examples, cite verifiable facts, and directly address the user's stated goal." Concrete criteria produce consistent scores; abstract criteria produce inconsistent ones.
Select and Configure a Judge Model
The judge model should be at least as capable as the agent being evaluated, and ideally more capable. Using a weaker model as a judge produces unreliable scores because the judge cannot distinguish between correct and incorrect reasoning on tasks it cannot solve itself. In practice, most teams use GPT-4o, Claude Sonnet or Opus, or Gemini 1.5 Pro as judge models regardless of which model their agent uses.
Set the judge model's temperature to 0 (or as low as the API allows) to maximize scoring consistency. Higher temperatures introduce random variation in scores that makes it impossible to detect real quality changes. With temperature 0, the same input and rubric produce the same score on repeated evaluations, which is essential for regression testing.
Configure the judge to return structured output: a numerical score for each rubric dimension plus a brief explanation of the reasoning. The explanation serves two purposes. First, it makes the score debuggable because you can read why the judge scored a 2 and decide whether you agree. Second, chain-of-thought reasoning before the final score improves scoring accuracy because the model reasons through the criteria before committing to a number.
Build the Judge Prompt with Structured Output
The judge prompt has four components: context (what the agent was asked to do), the agent's output (what it produced), the rubric (how to score it), and output instructions (what format the score should take).
A proven prompt structure is: "You are evaluating an AI agent's response. The user asked: [input]. The agent responded: [output]. Score the response on the following criteria using a 1-5 scale: [rubric with anchors for each level]. Return your evaluation as JSON with fields: score (integer 1-5), reasoning (1-2 sentences explaining the score)."
If you have a reference answer (the ideal response), include it in the prompt: "A good answer would include: [reference]. The agent's actual response was: [output]. Score how well the agent's response compares to the ideal." Reference-based judging produces more consistent scores because the judge has a concrete standard to compare against rather than relying entirely on its own knowledge.
For multi-dimensional evaluation, ask for a separate score on each dimension in a single call rather than making multiple calls. This is cheaper and often more consistent because the judge considers all dimensions together: "Return JSON with fields: accuracy_score, completeness_score, clarity_score, accuracy_reasoning, completeness_reasoning, clarity_reasoning."
Run Calibration Against Human Ratings
An LLM judge is only as trustworthy as its agreement with human judgment. Before relying on judge scores for decisions, you must validate that the judge and humans agree on what constitutes good versus poor quality for your specific use case.
The calibration process: select 50 to 100 representative agent outputs spanning the full quality range (some excellent, some mediocre, some poor). Have two or more human raters score each output using the same rubric you gave the judge. Run the LLM judge on the same outputs. Calculate agreement metrics: Pearson correlation for continuous scores, Cohen's kappa for categorical ratings, and percentage of cases where the judge and human median differ by more than one point.
Target at least 80% agreement (judge and human within one point on a 5-point scale) before using the judge in production. If agreement is below 80%, the most common fixes are: making rubric criteria more specific, adding more score-level anchors with examples, splitting vague dimensions into multiple precise ones, or switching to a more capable judge model. Rarely is the issue the judge model itself; almost always it is ambiguity in the rubric that humans interpret differently from the model.
Recalibrate periodically (monthly or after judge model updates) because model behavior changes between versions and your evaluation needs evolve as your agent improves. A judge calibrated against early-stage outputs may not discriminate well between good and excellent outputs once your agent's baseline quality improves.
Implement Bias Checks and Mitigations
LLM judges have systematic biases that can produce misleading scores if not addressed. The three most common biases are position bias, length bias, and self-preference bias.
Position bias: When comparing two outputs, the judge tends to favor whichever is presented first (or sometimes last). Test for this by running pairwise comparisons in both orderings and measuring whether scores change. Mitigate by averaging scores from both orderings, or by scoring outputs individually rather than comparatively.
Length bias: Judges tend to score longer responses higher than shorter ones even when the shorter response is more concise and accurate. Test for this by comparing judge scores on outputs of varying length that humans rate equally. Mitigate by adding explicit instructions like "brevity is acceptable when the answer is complete; do not penalize concise responses or reward unnecessary verbosity."
Self-preference bias: A model used as a judge tends to rate outputs from the same model family higher than outputs from competing models. If your agent uses GPT-4 and your judge uses GPT-4, the judge may be biased in favor of the agent's outputs. Test for this by having the judge evaluate outputs from multiple models and checking for systematic score differences. Mitigate by using a different model family for judging than for generation, or by validating against human scores specifically on cases where self-preference might occur.
Additional mitigations that improve judge quality: include "think step by step" instructions to force reasoning before scoring, randomize the order of any multiple-choice options, use odd-numbered scales (1-5 rather than 1-4) to allow a neutral midpoint, and explicitly instruct the judge to use the full range of scores rather than clustering everything around 3-4.
Deploy in Your Evaluation Pipeline
Once calibrated and bias-checked, integrate the judge into your evaluation pipeline. The typical integration points are: CI/CD (score agent outputs after every code change), batch evaluation (score a curated test set on a schedule), and production sampling (score a percentage of real production responses for ongoing quality monitoring).
Set quality thresholds based on your calibration data. If your calibration showed that human-rated "acceptable" outputs typically score 3.5+ from the judge, use 3.5 as your deployment gate. If average judge scores on your test set drop below this threshold after a change, block the deployment and investigate.
Monitor judge behavior over time. Track the distribution of scores, average scores per category, and score variance. A sudden shift in score distribution, even without a change to your agent, may indicate that a judge model update changed scoring behavior. When this happens, recalibrate immediately.
Budget for judge costs. Each judge evaluation requires an LLM API call, typically 500-2000 input tokens (rubric + output) and 100-300 output tokens (score + reasoning). At GPT-4o pricing, this runs approximately $0.002 to $0.005 per evaluation. For a test suite of 500 cases run daily, that is $1-2.50 per day. For production sampling at 10% of traffic on a 10,000 request per day agent, that is $20-50 per day. Factor this into your evaluation budget alongside compute and storage costs.
LLM judges are only as good as their rubrics. Invest your time in writing specific, concrete, example-rich evaluation criteria, then validate against human ratings before trusting the scores. A well-calibrated judge with a clear rubric is more reliable than a poorly-instructed human rater.