Testing AI Agents: Principles, Criteria, Metrics

What Does Testing AI Agents Mean?

Testing AI agents means verifying a system that plans autonomously, calls tools and may take different solution paths for the same input. An agent is more than a language model behind a prompt: it pursues a goal across multiple steps, decides on its next action along the way, calls APIs, databases or other systems, and reacts to their responses. That autonomy is what makes agents useful, and it is also what makes them the hardest test object the discipline has met so far.

The test object is not the individual output but the behaviour: the chain of planning, tool use and decisions, observed across many runs. Signing off an agent the way you would sign off a form with three input fields means testing past the actual risk.

Why Classic Test Cases Fall Short

Classic test cases compare a result against an expected value; for an AI agent, that single expected value often does not exist. The same request can be handled correctly along three different paths, and the same path can work today and fail tomorrow. A green test run therefore proves little, and a red one is not automatically a defect.

Then there is the input space. An agent that works with people in natural language receives rephrasings, typos, missing details, contradictory wishes. No enumerated set of test cases covers that space.

The consequence is a shift in thinking: evaluation instead of assertion. First define what must always hold, regardless of the solution path. Such invariants are strictly checkable: no action without authorisation, no invented prices, no answers outside the permitted scope. Everything beyond that, meaning how well the agent solves its task, is assessed statistically, across many runs and against defined thresholds. How systems without a clear test oracle can still be tested systematically is also the territory of metamorphic testing: instead of single expected values, it checks relations between related inputs.

Testing Agents vs. Testing with Agents

Testing agents and testing with agents are two different disciplines that currently share the label agentic testing. In testing with agents, the agent is the tool: it generates test cases, executes tests, repairs test scripts, a chapter of test automation. In testing agents, the agent is the test object, and that is the discipline this page covers. Anyone researching agentic testing should first settle which of the two directions they mean; the answers have little in common.

Principles for Testing AI Agents

Six principles carry the testing of AI agents. They do not replace test craft; they reorganise it.

1. Evaluate behaviour, not single outputs. What gets assessed is the space of possible outputs, not one output. A test case describes a task plus an acceptable corridor of behaviour, not one exact target state.
2. Invariants before quality. Check first what must never happen: unauthorised actions, violated business rules, output outside the permitted scope. These guardrails can be verified deterministically and belong in every run. How well the task is solved is the second question, never the first.
3. Repetition is part of the test case. A single run is a sample, not a statement. Only the distribution across many runs shows whether a result was skill or luck.
4. Tool calls are a test level of their own. Which API the agent calls with which parameters is observable and can be judged deterministically. Classic testing keeps working at this level, and it is the most stable basis for regression checks.
5. Observability before judgement. Without access to plans, intermediate steps and tool calls, you can detect misbehaviour but not diagnose it. Traces are to agents what logs are to servers.
6. Calibrate automated evaluation against humans. When a language model grades the agent’s answers (LLM-as-a-judge), that grading needs regular checks against human judgement. An uncalibrated judge measures precisely, and precisely beside the point.

Criteria: What to Examine in an AI Agent

The evaluation criteria for AI agents fall into six groups: task success, rule compliance, robustness, tool use, efficiency and transparency.

Task success asks whether the agent reaches the business goal, and reaches it completely: a half-cancelled order is not partial success, it is a defect. Rule compliance verifies adherence to business rules, permissions and compliance constraints in every single step. Robustness measures how stable the agent stays under rephrasings, incomplete information and deliberate manipulation, including adversarial input such as prompt injection. Tool use judges whether the agent calls the right tools with the right parameters and copes with their failures, say an API timeout. Efficiency counts steps, tokens, cost and latency per task; an agent that resolves every request correctly but needs forty tool calls to do it will not survive production. Transparency, finally, asks whether decisions can be traced and whether the agent recognises when a task must go to a human.

Metrics for AI Agent Reliability

The most important number about an AI agent is not its best result but the spread of its results. The central metrics follow from that:

• Task success rate: the share of runs in which the task was completed fully and within the rules.
• Consistency (pass^k): the probability that the same task succeeds in all k repetitions. The τ-bench benchmark established this metric, and its findings are sobering: the best agent tested there solved a good 60 percent of tasks in the retail domain on a single attempt; when the same task had to succeed eight times in a row, roughly 25 percent remained.
• Rule violation rate: breaches of invariants and guardrails per number of runs. The target is zero, and every breach gets analysed individually.
• Hallucination rate: the share of runs containing invented facts, invented sources, or actions claimed but never executed.
• Tool error rate: wrong tool choice, wrong parameters, unhandled failures of the systems being called.
• Escalation rate: the share of cases the agent correctly hands to a human, together with the counter-check: cases it should have handed over and did not.
• Cost and latency per task: the operational numbers on which agent projects fail in practice, long before quality becomes the problem.

Acceptance criteria then no longer read “all tests green” but as thresholds on distributions: success rate above x percent across n runs, zero rule violations, escalation rate inside a defined corridor.

A Five-Phase Approach to Testing AI Agents

Testing an AI agent follows five phases: specify the behaviour, design scenarios, define the evaluation, run repeatedly, keep measuring in production.

1. Specify the behaviour. Write down task scope, invariants and escalation rules. This specification is the test basis, and in most agent projects it simply does not exist.
2. Design scenarios. Realistic tasks from the target process, plus edge cases, incomplete information and deliberate attacks. For conversational agents, simulated users played by a second model have proven their worth.
3. Define the evaluation. Deterministic checks for invariants and tool calls, model-based grading for answer quality, human review samples for calibration. Metamorphic relations fill in where no expected value exists.
4. Run repeatedly and measure. Execute each scenario n times, collect distributions instead of single values, compute consistency metrics.
5. Keep measuring in production. Model versions change, prompts get tuned, data drifts. The evaluation suite therefore reruns on every change, as a regression test in the pipeline, and selected metrics keep running against live traffic.

Testing Multi-Agent Systems

In multi-agent systems, the critical failures emerge between the agents, not inside them. Every single agent can fulfil its role and the overall system still fails: through endless loops of mutual calls, contradictory intermediate states, tasks that vanish in handovers, emergent behaviour nobody designed.

Verification therefore needs two levels. At the level of the individual agent, the principles and metrics above apply. At the system level, questions of its own come in: does the collaboration terminate, or can agents keep each other busy forever? Do shared states stay consistent? Is it clear at every step which agent owns the task? Anyone who knows integration testing in classic systems will recognise the pattern: interfaces are where things break. Except that here, the interfaces are conversations.

Standards and Orientation

There is no established standard yet specifically for testing agentic systems, but two sources offer usable orientation. ISO/IEC 25059 extends the ISO/IEC 25010 quality model with AI-specific characteristics such as probabilistic behaviour, explainability and fairness; that provides a shared vocabulary for evaluation criteria that auditors and procurement understand as well. The ISTQB syllabus Certified Tester AI Testing (CT-AI), in version 2.0, describes test approaches for AI-based systems including generative AI, from data quality and model metrics to red teaming.

Both are maps, not recipes. Which criteria and thresholds apply to a concrete agent is decided by its context: a research assistant tolerates a different error rate than an agent that triggers payments.

Limits: What Testing Cannot Do for AI Agents

Testing does not make an AI agent deterministic; it makes the agent’s risk measurable and therefore decidable. Residual risk always remains, because no scenario catalogue covers an open input space, and every new model version shifts the behaviour again.

Dealing with that is an architecture question, not purely a testing question: narrowly scoped permissions, approval steps for critical actions, limits on cost and reach, monitoring in production. The larger the possible damage of a single agent action, the less a quality strategy may rely on pre-release testing alone. That is not capitulation; it is the same logic the industry has applied to complex systems for decades: verify what can be verified, and contain what cannot.

Support for Testing AI Agents

Richard Seidl is a consultant and coach for software quality and software testing based in Germany who supports companies in testing AI systems and AI agents: from test strategy and evaluation criteria to building the evaluation pipeline. He works vendor-neutral, sells no test tool and is tied to no framework. Behind that stand more than 25 years in software testing, eight books on the subject, and work with teams from mid-sized companies to large enterprises; in 2025 he received the German Software Quality Award (Deutscher Preis für Software-Qualität).

The entry point is usually small: a workshop that produces invariants, criteria and a first set of metrics for one concrete agent. From there, the evaluation pipeline grows with the system.