How to evaluate AI agents before production

01

Start with a task contract

Before choosing metrics, define what the agent is responsible for and what remains outside its authority. List allowed tools, required evidence, stop conditions, escalation rules, and the observable result of success.

A vague goal such as “help customers” cannot be evaluated consistently. A task contract such as “classify the request, retrieve the approved policy, draft a cited answer, and escalate uncertain cases” creates testable behavior.

02

Build a representative evaluation set

Use anonymized examples from real work: routine cases, incomplete inputs, conflicting evidence, tool failures, malicious instructions, permission boundaries, and unusual but valid requests. Label the expected outcome and the actions that must never occur.

• Normal successful tasks
• Ambiguous or underspecified requests
• Unavailable tools and timeout scenarios
• Conflicting sources and stale information
• Requests that require human approval
• Attempts to override system policy

03

Score the trajectory

A trajectory is the sequence of observations, decisions, tool calls, and outputs. Score tool selection, argument accuracy, evidence use, policy compliance, recovery, stopping behavior, and final usefulness.

Automated graders can accelerate review, but important release decisions need calibrated human examples. Regularly compare grader scores with expert judgment and investigate disagreement.

04

Measure operational performance

Offline accuracy does not prove business value. In production, track completion rate, escalation rate, correction rate, time saved, user outcome, cost per completed task, latency, and incidents.

Segment metrics by task type and risk level. A high average score can hide serious failures in a small but important class of requests.

05

Create release and rollback gates

Define minimum thresholds before testing begins. Require zero critical permission violations, acceptable recovery from tool errors, a maximum cost envelope, and human sign-off for high-risk workflows.

Version the prompt, model, tools, policies, and evaluation set together. Maintain a rollback path so a regression can be contained without rebuilding the system.