LLM as a Judge: Reproducible Agent Evaluation

An LLM can review thousands of agent responses faster than your team can. That does not make its verdict correct. A useful LLM-as-a-judge pipeline needs observable criteria, versioned test cases, complete traces, and human validation. Otherwise, a formatting change can look like an improvement while factual errors pass unnoticed. This guide builds an evaluation workflow you can reproduce, audit, and use as a release gate.

What LLM as a Judge Evaluates—and What It Cannot Prove

LLM as a judge is an evaluation pattern in which one model scores, selects, or ranks generated responses. It is often reference-free, although it can also evaluate outputs against a rubric, instructions, or a reference answer (source).

That flexibility helps with properties that are difficult to express as exact assertions:

• Did the agent answer the requested question?
• Is the explanation supported by the supplied evidence?
• Did it follow the user's constraints?
• Was its tool choice reasonable?
• Does the final answer reflect the tool results?
• Is the response clear enough to act on?

A judge does not prove that an answer is true. It produces another model output. Bias, randomness, and unclear ground truth remain problems, especially for subjective tasks (source).

Treat the verdict as a measurement, not an oracle. Keep deterministic checks for anything software can verify directly. Use an LLM only where semantic interpretation is necessary.

For example, a lead-enrichment agent might return:

{
  "company": "Example Corp",
  "domain": "example.com",
  "employee_band": "unknown",
  "evidence_url": "https://example.com/about"
}

Your test code can validate the schema, allowed values, and URL presence. A judge can assess whether the evidence supports the generated summary. Neither should silently replace the other.

Design an Observable Rubric for Agent Responses and Actions

“Is this a good answer?” is not a reproducible evaluation prompt. Different runs can interpret “good” differently.

Break quality into criteria with observable evidence:

Criterion	What the judge inspects	Example failure
Task completion	Requested deliverables in the final answer	The agent omits the requested report
Factuality	Claims compared with trace evidence	A summary contradicts an API response
Instruction following	User and system constraints	The agent uses a forbidden tool
Tool selection	Tool suitability for the task	It guesses instead of querying available data
Argument validity	Tool-call arguments and types	A date field contains free-form prose
Constraint compliance	Allowlist, scope, and output rules	A monitoring agent visits an unapproved domain
Side effects	Actions visible in the trace	The agent creates duplicate records
Safety	Bounded behavior and escalation	An irreversible action runs without approval

Use categorical verdicts such as pass, partial, fail, and abstain. Require the judge to quote evidence from the trace. Add positive and negative examples to each criterion.

An abstain result matters. It distinguishes “the agent failed” from “the available evidence cannot support a verdict.”

A compact rubric file could look like this:

rubric_version: rubric-agent-response-v1
criteria:
  task_completion:
    question: Did the agent produce every requested deliverable?
    pass_example: The report contains the requested summary and evidence URLs.
    fail_example: The report contains a summary but no evidence URLs.
  factuality:
    question: Does each material claim match the supplied tool evidence?
    pass_example: The summary restates the API result without adding claims.
    fail_example: The summary invents a status absent from the API result.
  side_effects:
    question: Did the agent perform only permitted actions?
    pass_example: The trace contains a read operation on an allowed resource.
    fail_example: The trace contains an unapproved write operation.
allowed_verdicts:
  - pass
  - partial
  - fail
  - abstain

Run deterministic checks first. Schema validity, exact values, reference answers, tool names, and argument types do not need semantic judgment. Pass their results into the judge so it cannot reinterpret a failed assertion as acceptable.

Build a Versioned Evaluation Dataset Before Running the Judge

Do not start by collecting arbitrary production conversations. Define the dataset as code.

Each case should contain:

• A stable case ID.
• The task and input.
• Permitted tools.
• Expected outcome.
• Reference evidence.
• Difficulty and risk labels.
• Agent version.
• Expected deterministic assertions.
• Links to any fixture data.

Cover normal requests, ambiguous inputs, tool failures, multi-step tasks, and adversarial cases. For an API-backed reporting agent, include expired credentials, empty results, malformed responses, and conflicting evidence. For a monitoring agent, test its allowlist and its behavior when a page cannot be retrieved.

Store cases beside the evaluator:

evaluation/
├── cases/
│   ├── report-empty-result.yaml
│   ├── report-tool-timeout.yaml
│   └── monitor-disallowed-domain.yaml
├── rubrics/
│   └── agent-response.yaml
├── schemas/
│   └── judge-result.json
└── run_evaluation.py

Version the dataset, agent prompt, rubric, judge prompt, and judge model together. A score without those inputs cannot be reproduced.

The broader reproducible agent evaluation workflow explains how reference datasets, repeated trials, graders, and production feedback fit together.

Avoid editing an existing case to make a regression disappear. Create a new dataset version. Preserve the old result and document why the expectation changed.

Run a Reproducible LLM-as-a-Judge Pipeline on Full Agent Traces

A final answer alone hides most agent failures. Capture the complete evaluation snapshot:

• System and user prompts.
• Tool definitions.
• Tool calls and arguments.
• Intermediate tool results.
• Final response.
• Agent and model identifiers.
• Generation parameters.
• Timestamps.
• Dataset and rubric versions.
• Deterministic check results.

Force the judge to return structured JSON. Reject malformed output instead of extracting a verdict from prose.

{
  "case_id": "report-tool-timeout",
  "rubric_version": "rubric-agent-response-v1",
  "overall_verdict": "fail",
  "criteria": {
    "task_completion": {
      "verdict": "fail",
      "evidence": "The trace ended after the tool timeout.",
      "failure_label": "unhandled_tool_failure"
    },
    "factuality": {
      "verdict": "abstain",
      "evidence": "No source data was returned.",
      "failure_label": null
    }
  },
  "confidence": "high"
}

The following dependency-free script validates the result shape and converts criterion verdicts into a release decision:

#!/usr/bin/env python3
import json
import sys
from pathlib import Path

ALLOWED = {"pass", "partial", "fail", "abstain"}

def validate(result):
    required = {"case_id", "rubric_version", "overall_verdict", "criteria"}
    missing = required - result.keys()
    if missing:
        raise ValueError(f"Missing fields: {sorted(missing)}")

    if result["overall_verdict"] not in ALLOWED:
        raise ValueError("Invalid overall verdict")

    for name, item in result["criteria"].items():
        if item.get("verdict") not in ALLOWED:
            raise ValueError(f"Invalid verdict for {name}")
        if not item.get("evidence"):
            raise ValueError(f"Missing evidence for {name}")

    return result["overall_verdict"] not in {"fail", "abstain"}

result = json.loads(Path(sys.argv[1]).read_text())
accepted = validate(result)
print(json.dumps({"case_id": result["case_id"], "accepted": accepted}))
sys.exit(not accepted)

Run it with:

python evaluation/validate_judgment.py evaluation/results/report-tool-timeout.json

Repeat selected cases to expose unstable verdicts. For pairwise comparisons, randomize candidate positions. A cited evaluation methodology uses this counterbalancing to reduce position effects (source).

Also test non-semantic sensitivity. Start with the same candidate response, then inject factual errors, false references, emojis, Markdown, or more polished formatting. The Attack Successful Rate measures how often a perturbation changes the judge's preference. Some reported attacks exceeded 50% ASR in certain cases (source).

That is why appearance must not stand in for correctness. You can incorporate these checks into a pre-production agent comparison workflow.

Validate the Judge Against Humans—Especially False Positives

The most dangerous error is often a false positive: the judge passes an agent output that a reviewer would reject.

Build a review sample from:

• High-risk tasks.
• Borderline verdicts.
• Judge abstentions.
• Disagreements across repeated runs.
• Randomly selected passes and failures.

Have independent reviewers label the same cases without seeing the judge's verdict. Resolve disagreements through adjudication. Then compare the final human labels with the automated decisions.

For pass/fail outcomes, build a confusion matrix and track:

• Precision.
• Recall.
• False-positive rate.
• False-negative rate.
• Confidence calibration.
• Agreement between human annotators.
• Agreement between the judge and adjudicated labels.

The goal is not to assume absolute correctness. It is to measure how closely the judge agrees with qualified human annotation. An Alternative Annotator Test can help assess whether a model can replace human annotation for a defined task and at what cost (source).

When reviewers confirm a judge error, do not overwrite the original score. Preserve it. Add the case to the dataset, improve the rubric examples, and adjust the decision rule in a new version.

Operate the Judge as a Regression Gate, Not a Single Score

A single average can hide a severe failure. Gate releases by criterion and risk tier instead.

A candidate agent might improve task completion while introducing unsafe tool calls. An aggregate score could still rise. A criterion-level gate blocks that release and shows why.

For every comparison:

1. Run both agent versions against identical case IDs.
2. Apply deterministic assertions before semantic grading.
3. Preserve raw judge responses and evidence.
4. Review changed, unstable, and high-risk verdicts.
5. Record accepted exceptions.
6. Publish the next dataset version.

Monitor judge-model changes separately from agent changes. A new judge can move scores even when the agent is identical. Track model identifiers, latency, token usage, and disagreement across repeated cases.

Keep mandatory human review for irreversible actions and high-impact failures. No semantic score should authorize an action merely because it exceeds a threshold.

Traces, datasets, evaluations, deployment, and monitoring should form one continuous reliability loop. The LangSmith reliability loop guide shows how those components connect in practice.

LLM as a judge works best as one evaluator in a layered system. Deterministic assertions catch exact failures. The judge evaluates semantic quality. Humans calibrate both. Once that pipeline runs with your deployed agent jobs, every release produces evidence you can inspect—not just a score you have to trust.