Skip to content
AI & Automation7 min read

AI Agent Evaluation: How to Test Accuracy, Reliability and Performance

How to evaluate AI agents: building evaluation datasets, task success, tool-call accuracy, groundedness, policy compliance, latency, cost, LLM-as-judge, regression testing and production evaluation.

01

Quick answer

Evaluate AI agents on whole tasks, not single answers. Build a dataset of real cases with expected outcomes, run the agent against it, and score the final result (task success, correct end state), the trajectory (tool choice, argument validity, step count, policy compliance), output quality (groundedness, tone) and operations (latency, cost per task). Use deterministic checks wherever possible, LLM-as-judge with human calibration where needed, gate every release on regression results and keep scoring sampled production runs.

02

Where This Fits

Evaluation needs traces from agent observability, informs the thresholds in human-in-the-loop design and tests the controls in guardrails. For retrieval-specific metrics, see RAG.

Evaluation of individual models and prompts, rather than whole agents, is covered in AI model evaluation.

03

What to Measure

LevelMetricHow to score
OutcomeTask successCompare end state or output with expected result
OutcomeEscalation correctnessDid it hand off when it should, and only then?
TrajectoryTool-call accuracyRight tool, valid and correct arguments
TrajectoryEfficiencySteps, loops, redundant calls
QualityGroundednessClaims supported by sources or tool results
SafetyPolicy complianceNo forbidden actions or disclosures
OperationsLatency and cost per taskMeasured per run
04

Building an Evaluation Dataset

Start from real cases, anonymized where needed. Include typical cases, known edge cases, cases where the right answer is to escalate, and adversarial inputs such as prompt injection attempts. For each case record the input, relevant context or system state, the expected outcome and, where useful, expected tool calls. Label who agreed the expected outcome. Keep a held-out set you do not tune against.

  • Typical cases weighted by real volume
  • Edge cases from support tickets and incident reports
  • Should-escalate cases
  • Adversarial and malformed inputs
  • Cases that exercise every tool and policy
  • A held-out set for honest comparison
05

Scoring Methods

Prefer deterministic checks: did the order status become 'cancelled'? Does the output match the schema? Was the refund under the limit? Use reference comparison for extracted fields. Use LLM-as-judge for qualities such as helpfulness or groundedness, with a written rubric, and calibrate it by comparing its scores with human ratings on a sample. Use human review for high-stakes or ambiguous cases.

06

Evaluating Tool Use and Trajectories

For agents, the path matters. An agent that reaches the right answer by calling a forbidden tool has failed. Score each step: was the tool appropriate, were arguments valid, was the call necessary? Compare against expected calls where the path is fixed, and against state checks where several paths are acceptable. Flag loops and excessive steps, which drive cost and latency.

Shipping agent changes without knowing what they break?

ZSpace Labs sets up evaluation datasets, scoring and release gates so every prompt, model or tool change is tested against real cases.

Start a Project
07

Regression Testing and Release Gates

Run the evaluation set on every change to prompts, tools, retrieval or model version, and compare against the current production version. Define gates: overall success must not drop, safety failures must be zero, cost and latency must stay within budgets. Track results by case type, because averages hide regressions in small but important categories.

Rollout strategies after the gate, such as shadow tests and canaries, are covered in AI application release management.

08

Production Evaluation

Offline sets never capture everything. In production, score a sample of live runs automatically, collect user and reviewer feedback, watch outcome signals (corrections, escalations, complaints) and review failures weekly. Turn every confirmed failure into a new test case. Store traces with privacy controls, because they contain customer data.

09

Tools for Agent Evaluation

Options include evaluation features in model providers' platforms, open-source frameworks, observability platforms with evaluation built in, and simple custom harnesses that replay cases and score results. Choose tools that store datasets with versions, run on every change, and link scores to traces so failures can be debugged.

10

Advantages and Limitations of Evaluation Approaches

MethodStrengthsLimitations
Deterministic checksExact, cheap, repeatableOnly for checkable outcomes
Reference comparisonClear for extraction tasksNeeds labelled answers
LLM-as-judgeScales to open-ended qualityBias, needs calibration
Human reviewMost trustedSlow and costly
Production signalsReal behaviourLagging and noisy
11

How to Set Up Evaluation Step by Step

  • 1. Agree success criteria with the process owner
  • 2. Collect and label 50 to 200 real cases
  • 3. Write deterministic checks for end states and schemas
  • 4. Add rubric-based judging for open-ended quality, calibrated against humans
  • 5. Run a baseline and record results by case type
  • 6. Add the run to CI as a release gate
  • 7. Sample production runs and feed failures back into the set
12

Anatomy of an Evaluation Case

A good evaluation case records enough to replay the task and judge it objectively. Store cases as versioned data alongside the code they test.

Example: one agent evaluation case (illustrative)
{
  "id": "refund-017",
  "input": "I was charged twice for order ORD-104233, please fix it",
  "setup": { "order": "ORD-104233", "payments": ["pi_a", "pi_b"], "delivered_days_ago": 3 },
  "expected": {
    "outcome": "refund_issued",
    "refund_amount": 59.90,
    "tools_required": ["get_order", "list_payments", "refund_payment"],
    "tools_forbidden": ["cancel_order"],
    "must_escalate": false
  },
  "checks": ["end_state_refund_count == 1", "reply_mentions_refund_timeline"],
  "labelled_by": "support-lead",
  "tags": ["refund", "duplicate_charge"]
}
13

Evaluating Retrieval Inside Agents

Many agents depend on retrieval for policies or records. When an agent gives a wrong answer, check whether it retrieved the right sources before blaming the model. Add retrieval checks to evaluation cases (which sources should appear?) and measure faithfulness of the final answer to what was retrieved. The RAG-specific metrics are covered in the RAG guide.

14

Who Owns Evaluation

Evaluation works when ownership is clear. The process owner agrees what correct looks like and approves labels; engineers maintain the harness, CI gates and production sampling; reviewers or QA staff label new cases from production failures. Review results together after each release and monthly for trends.

15

Worked Example

An illustrative scenario, not a client case: a support agent passes informal testing, but an evaluation set of 250 real tickets shows it issues refunds on orders outside the return window in a small share of cases. The fix is a policy check inside the refund tool, not a prompt change. The case is added to the regression set, and the release gate now requires zero policy violations.

16

Common Mistakes

  • Testing with a handful of invented examples
  • Scoring only the final answer, not the actions
  • Trusting an uncalibrated LLM judge
  • Averages that hide failures in important case types
  • No evaluation when the model provider updates a model
  • Not adding production failures to the test set

Want evidence that your agent is ready for production?

Talk to ZSpace Labs about AI evaluation and agent development.

Start a Project
17

Production Sign-Off: What Must Pass Before Go-Live

Before an agent handles real cases unsupervised, agree pass thresholds across every dimension, not just answer quality. A practical sign-off covers:

DimensionWhat to checkExample threshold
Task successShare of test cases completed correctly end to endAgreed target per case type
Factual accuracyAnswers grounded in approved sourcesNo unsupported claims on policy or price
Tool-call accuracyRight tool, right arguments, no unnecessary callsWrong-tool rate below target
ReliabilitySame input gives acceptable output across repeated runsConsistent on repeated runs
Latency and costTime and cost per completed taskWithin budget at expected volume
Security and permissionsInjection attempts, out-of-scope requests, permission boundariesNo unauthorized actions
Failure recoveryBehaviour when tools fail or data is missingEscalates with context; no loops
EscalationHands off the right cases to peopleNo high-risk case handled alone
RegressionFull test set re-run on every model, prompt or tool changeNo drop beyond agreed tolerance

Pro tip

Pair this sign-off with the autonomy level you plan to grant: stricter thresholds for higher autonomy. See how much autonomy to give an AI agent and how to stop agents looping and overspending.

18

Conclusion

Agent evaluation is how you earn the right to automate. Build datasets from real cases, score outcomes and trajectories, gate releases and keep learning from production. Related: observability, guardrails and agent development.

Run these evaluations in an isolated environment with mock tools before production access (AI agent sandbox), and include unsupported-claim checks from reducing AI agent hallucinations.

FAQ

Common questions.

Measuring whether an agent completes tasks correctly, safely and efficiently, using test cases with expected outcomes, scoring of each step and the final result, and comparison across versions.

A chatbot is mostly judged on its answer. An agent must also be judged on its trajectory: which tools it called, with what arguments, in what order, whether it respected policies and whether the end state in other systems is correct.

Enough to cover your main case types and known edge cases with several examples each. Many teams start with 50 to 200 real cases and grow the set from production failures.

Using a language model to score outputs against criteria such as correctness or groundedness. It scales evaluation but must be checked against human judgements, and it should not be the only measure for critical decisions.

Whether the agent chose the right tool with valid, correct arguments at each step, compared with expected calls or validated by checking the resulting state.

Whether claims in the output are supported by the sources or tool results the agent had, rather than invented.

Before every release that changes prompts, tools, models or retrieval, on a schedule against production samples, and after incidents to add new test cases.

Yes, through sampled scoring of live runs, user feedback, reviewer decisions and outcome metrics such as escalations and corrections, with privacy controls on stored data.

It depends on the cost of errors and the fallback. A drafting assistant reviewed by people can launch with a lower rate than an agent acting automatically. Agree the threshold with the process owner before testing.

Get in touch

Have a project in mind?

Whether you're building a new digital product, improving an existing website, or looking to automate part of your business — let's talk.