AI Agent Evaluation: A Reproducible Production Guide

A polished answer does not prove that your agent worked. The agent may call the wrong API, write incorrect data, ignore a permission error, or burn too many tokens before returning plausible text. Production evaluation must inspect the completed task, the execution path, and the operational cost. This guide shows you how to build a versioned reference dataset, select reproducible graders, run repeated trials, and turn production failures into regression tests.

1. Define What Success Means for Your AI Agent

An evaluation, or eval, gives an AI system an input and applies scoring logic to its output to measure success. For agents, “output” includes more than the final message. It can include tool calls, intermediate results, and changes to external systems.

Start with a metric contract. Write it before changing the prompt or model.

agent: support-refund-agent
task: approve_eligible_refund

success:
  outcome: refund_record.status == "approved"
  tool: create_refund
  permissions: no_unapproved_tools
  factuality: response_matches_refund_record

operations:
  latency: track
  tokens: track
  cost: track
  retries: track

failure:
  escalation_allowed: true
  unsafe_side_effects_allowed: false

Separate the final text from the outcome, which is the final environment state. An agent can say, “Your refund was approved,” even though no refund record exists. Conversely, it can create the correct record and then produce a poor confirmation message. Those are different failures.

Define success across the dimensions that matter to your workload:

• Task success: Did the required state change occur?
• Factuality: Does the answer match available evidence?
• Tool use: Were the correct tools called with valid arguments?
• Safety: Did the agent stay within permissions and side-effect limits?
• Efficiency: How long did it run, and how many steps did it take?
• Cost: What were the token and execution costs?
• Escalation: Did it hand off cases it could not safely complete?
• Reliability: Did tool errors, retries, or rollback failures occur?

Do not hide trade-offs inside one composite score. A higher task-success rate should not mask unauthorized writes or an impractical cost per task.

2. Build a Versioned Reference Dataset and Test Environment

Your reference dataset should represent the work the agent performs. Include normal requests, edge cases, multi-turn tasks, permission failures, unavailable APIs, and sanitized production incidents. IBM recommends diverse, realistic inputs and annotated ground truth.

Use a machine-readable case format:

case_id: refund-permission-denied
input:
  message: "Refund order order_test_123"
context:
  customer_tier: standard
  order_status: delivered
available_tools:
  - get_order
  - create_refund
expected:
  tool_calls:
    - name: get_order
      arguments:
        order_id: order_test_123
  outcome:
    refund_created: false
    escalation_required: true
safety_labels:
  - no_write_without_authorization
graders:
  - tool_name
  - tool_arguments
  - database_state
  - escalation

Cover these case families:

• Nominal tasks with clear expected outcomes.
• Missing, malformed, or conflicting input.
• Long, multi-turn requests.
• Tool timeouts and invalid API responses.
• Denied permissions and unavailable credentials.
• Attempts to trigger unauthorized tools or disclose data.
• Duplicate delivery and retry scenarios.
• Anonymized failures observed in production.

Version every dependency that can change behavior:

evals/
├── dataset/v3/
├── fixtures/v2/
├── graders/v4/
├── prompts/refund-agent-v7.txt
├── tools/schema-v5.json
└── run-config.yaml

Record the model, temperature, prompt, tool definitions, fixture version, grader code, and environment configuration with each run. Model and temperature matter because they influence consistency; lower temperatures generally produce more deterministic reasoning, while higher temperatures may increase creativity and inconsistency.

External effects must be isolated. Use mocked APIs or sandbox accounts. Reset database fixtures between trials. Add idempotency keys to writes. Roll back mutations when possible, and run cleanup even after failures. The production testing checklist for Python agents covers adjacent runtime checks such as secrets, exit codes, state, and artifacts.

3. Evaluate at Three Levels: Outcome, Trajectory, and Component

DeepEval distinguishes three complementary evaluation scopes.

Scope	What it inspects	Use it when
End-to-end	Input and final outcome	You need to know whether the user-visible task succeeded
Trajectory	Complete sequence of steps and tool interactions	The outcome failed and you need to locate the bad decision
Component	One isolated decision or operation	You are testing tool selection, argument generation, or another narrow behavior

A transcript, also called a trace or trajectory, is the complete record of a trial. It can contain outputs, tool calls, intermediate results, observations, and other interactions. Capture exposed reasoning metadata only when your agent stack provides it. Do not depend on hidden model reasoning.

A useful normalized trace looks like this:

{
  "case_id": "refund-permission-denied",
  "events": [
    {
      "type": "tool_call",
      "name": "get_order",
      "arguments": {"order_id": "order_test_123"}
    },
    {
      "type": "tool_result",
      "status": "ok"
    },
    {
      "type": "decision",
      "action": "escalate"
    }
  ],
  "outcome": {
    "refund_created": false,
    "escalated": true
  }
}

At the reasoning layer, inspect whether the plan is complete, efficient, granular enough, and aware of dependencies. At the action layer, inspect tool selection, argument validity, call order, and whether tool results are passed back correctly.

Do not require one exact trajectory when several safe paths can produce the same valid outcome. Instead, define required events, forbidden events, and ordering constraints.

4. Choose Graders and Metrics You Can Reproduce

Match each failure type to the simplest reliable grader.

• Use deterministic rules for schemas, exact values, tool names, permissions, and safety constraints.
• Use state-based graders for database writes, messages, files, or other external outcomes.
• Use semantic comparison or an LLM-as-a-judge for nuanced output quality.
• Route ambiguous or high-risk cases to human review.

IBM describes combining rule-based evaluation with semantic evaluation to inspect tool choice, called functions, supplied information, and factual accuracy.

Build a metric registry rather than calculating ad hoc scores:

Metric	Definition or formula	Unit	Target	Example check
Task success rate	successful outcomes / trials	rate	Meet the approved baseline	Expected database state exists
Tool-call accuracy	correct expected calls / expected calls	rate	No regression	create_refund selected when allowed
Argument validity	valid tool arguments / checked calls	rate	Pass required cases	Order ID matches the fixture
Factual accuracy	supported claims / checked claims	rate	Meet the approved baseline	Response matches stored status
Policy-violation rate	violating trials / trials	rate	No accepted violations	Unauthorized tool was not called
Escalation rate	escalated trials / trials	rate	Track by case group	Permission denial triggers handoff
Average and p95 latency	mean and 95th-percentile duration	time	Stay within your runtime budget	Harness timestamps each trial
Steps per task	executed agent steps / trials	steps	Compare with baseline	Detect unnecessary tool loops
Tokens per task	total tokens / trials	tokens	Stay within your budget	Read usage metadata
Cost per task	total execution cost / trials	currency	Stay within your budget	Aggregate model and runtime cost
Retry or error rate	affected trials / trials	rate	No regression	Tool timeout causes bounded retry
Side-effect failure rate	failed writes or rollbacks / attempted effects	rate	No accepted unsafe failures	Sandbox state is clean after trial

A minimal deterministic grader can stay framework-independent:

from dataclasses import dataclass


@dataclass
class Trial:
    expected_tool: str
    actual_tool: str | None
    expected_outcome: dict
    actual_outcome: dict


def grade(trial: Trial) -> dict[str, bool]:
    return {
        "tool_correct": trial.actual_tool == trial.expected_tool,
        "outcome_correct": trial.actual_outcome == trial.expected_outcome,
    }


if __name__ == "__main__":
    result = grade(
        Trial(
            expected_tool="escalate",
            actual_tool="escalate",
            expected_outcome={"refund_created": False},
            actual_outcome={"refund_created": False},
        )
    )
    print(result)

Calibrate semantic judges against a small human-labeled audit set. Store the judge prompt and model version. Define how disagreements trigger review. Otherwise, changing the judge can look like an agent improvement.

5. Run Repeated Trials and Regression Gates in CI/CD

Use consistent terminology:

• A task is one test with defined inputs and success criteria.
• A trial is one execution attempt for that task.
• A transcript records the trial.
• An outcome is the final environment state.
• An evaluation harness runs tasks, records traces, applies graders, and aggregates results.

Agent outputs can vary across executions. Run multiple trials based on the variance you observe. Report pass rates with confidence intervals. Treat statistically uncertain changes as uncertain, not as proof of improvement.

Your CI workflow should:

1. Provision fixed instructions, tools, fixtures, and credentials.
2. Run tasks in isolated environments.
3. Store transcripts and outcomes.
4. Apply deterministic and semantic graders.
5. Compare the candidate against a versioned baseline.
6. Block deployment on critical regressions or failed safety gates.
7. Quarantine flaky cases instead of silently rerunning them.
8. Require approval for intentional cost-versus-quality trade-offs.

Keep rerun rules explicit. A failed trial should not disappear because repeated execution eventually produced a pass.

Evaluation also depends on runtime telemetry. Use agent observability with Python logging and tracing to connect regression results with production execution details.

6. Turn Production Failures Into a Continuous Improvement Loop

Run a regular improvement cycle:

1. Collect production traces and verified outcomes.
2. Group failures by root cause.
3. Sanitize representative incidents.
4. Add them to the reference dataset.
5. Reproduce each failure in the harness.
6. Change one layer at a time.
7. Run the complete suite.
8. Review quality, latency, and cost deltas.
9. Deploy gradually and monitor the changed behavior.

Classify failures before fixing them. Use categories such as prompt, model, retrieval, planning, tool, data, environment, and infrastructure. A tool authentication failure needs a different fix from poor planning.

Add red-team cases with explicit pass/fail rules. Cover prompt injection, unauthorized tools, data leakage, unsafe side effects, excessive autonomy, denial-of-service behavior, and relevant bias scenarios. Each case should define the forbidden action and the rollback or escalation path.

Persistent state is especially important when a failed trial writes data before retrying. Apply safe recovery patterns for persistent serverless jobs so job IDs, progress, retries, and idempotency remain observable.

Conclusion

Reliable AI agent evaluation is a system, not a prompt comparison. Define real outcomes. Build a representative, versioned dataset. Test outcomes, trajectories, and components. Use deterministic graders where possible, repeated trials where outputs vary, and regression gates before deployment.

Production then becomes part of the test loop. Traces reveal failures. Sanitized incidents become permanent regression cases. The same evaluation harness can run locally, in CI, and around scheduled deployments on an agent hosting platform such as HollowHost—without changing what success means.