Why Impressive AI Demos Prove Little
An impressive demo says very little about how an AI reaches its results. AI evaluation metrics judge an AI system on many test cases at once instead of one showcase. Testing also works the other way round, with AI taking on testing tasks from performance testing to drafting test cases.
AI has been a research topic for decades, and in recent years it has made a very public victory lap. In 2016, AlphaGo beat the world-class player Lee Sedol at Go, a game whose course is much harder to predict than chess. Image recognition helps diagnose dangerous diseases early and monitors public spaces.
But how did the AI arrive at its result? In 2019, a team around Sebastian Lapuschkin showed that an image classifier recognized many horse pictures without looking at the horses at all. What tipped the balance was a source tag in the corner of the image, which appeared conspicuously often on the horse photos in the training data. Remove the tag and the horse was gone for the model; add it to a photo of a car and the car became a horse. It brings to mind “Clever Hans,” the horse that only appeared to count.
Exaggeration in Both Directions
The mood can flip just as fast. After an Uber test vehicle struck and killed a pedestrian in Arizona in 2018, many people saw self-driving cars as a danger. Yet in 2020, 140 people died in road accidents in the German state of Brandenburg alone, almost three a week. Even an AI that falls short of perfect could help there.
I see exaggeration in both directions. AI can achieve a lot, even in safety-critical applications, provided we can test and safeguard it properly. That is where the work for us testers begins.
What Do AI Evaluation Metrics Measure?
They measure how often a model gets things right across many cases, because an AI rarely gives a clear right or wrong. A single test in which the model mistakes a cat for a dog says little about the model as a whole. The ISTQB Certified Tester AI Testing (CT-AI) syllabus, released as version 2.0 in April 2026, calls for test suites large enough to give statistically significant results.
For classifiers, the basic tool is the confusion matrix. It sets prediction against reality: true positives and true negatives are the hits, false positives are false alarms, and false negatives are misses. The common metrics follow from it. Accuracy counts all correct classifications. Precision shows how many of the positive predictions were right, and recall how many of the actual positives the model found. The F1 score combines precision and recall as their harmonic mean.
How much each kind of error matters depends on the domain. If an AI diagnoses a tumor that isn’t there, that’s unpleasant. If it misses a tumor that is there, it can cost the patient’s life. In diagnostics, recall therefore counts for more than precision.
The same arithmetic applies when several AI systems compete. A comparison is only fair if every candidate gets the same test data and the same metrics. The syllabus names A/B testing and back-to-back testing for this, plus benchmark suites for generative AI. AI agents raise the bar again, because they plan on their own, call tools, and solve the same task a little differently each time.
The Training Data Is Part of the Test Object
The horse example shows what testers need to watch: an AI learns what is in the data, not what we meant. Evaluating AI therefore includes checking the training data. Which features could the model have picked up that have nothing to do with the actual task?
This has since become a test level of its own, input data testing. It covers the training data and the production data the system works with in operation. Is the data biased? Does it represent the situations the system will face? Are the labels correct? Does the data pipeline work?
Do Test Levels and White-Box Techniques Apply to AI?
Yes, with additions. When this article first appeared in 2021, that was an open question; CT-AI now answers it. The conventional test levels, from component testing to acceptance testing, still apply to every part of the system without AI and to how those parts work with the model. Two specialized levels come on top: input data testing and ML model testing, which checks whether the model meets its required performance criteria.
White-box techniques have found their place too. Neural networks don’t follow explicitly coded paths, so dedicated coverage measures such as neuron coverage show which parts of the network a test activates at all. High coverage doesn’t guarantee that the model generalizes, though. Neural networks can learn spurious correlations too and be right for the wrong reasons, just like the horse classifier with the source tag. That is why structural measures need techniques like adversarial testing and metamorphic testing alongside them.
Robustness and Security: Small Changes, Big Effects
A test checks more than whether the new algorithm beats the old one. Alongside functional and non-functional testing, IT security belongs in scope. Even small changes to a traffic sign can have consequences: if a self-driving car reads the “30” on a speed limit sign as “80,” it will race through town accordingly.
This isn’t a thought experiment. In 2020, security researchers at McAfee put a thin strip of tape on a 35 mph sign. The camera in an older Tesla read it as 85, and the cruise control sped up. Contradictory situations, such as a stop sign on the highway, are just as dangerous.
The syllabus calls these adversarial examples: inputs created by small perturbations of a working example that make the model produce a wrong result with high confidence. Adversarial testing hunts for such inputs so the team can harden the model against them.
When Should the System Be Allowed to Learn?
It depends on the application domain, and testing checks that the system stays within the chosen limit. An AI that keeps learning in operation specializes. A commuter’s self-driving car soon knows every quirk of the daily route and “forgets” the rest. The alternative is to restrict learning to maintenance or development.
It makes a big difference whether an AI-based system is locked or adaptive. A locked system doesn’t change its behavior after deployment, which makes it far easier to test. If it is retrained, it counts as a new system and needs a fresh round of testing. An adaptive system keeps learning in operation, so its future behavior can’t be fully predicted. An automated test suite for the core functions, run after every significant change, helps here. Drift testing also shows whether the input data or its meaning has shifted over time.
How Can AI Help with Software Testing?
Wherever patterns and similarities matter. Performance testing is an obvious field. Depending on the input data, the AI spots anomalies in system behavior and system load and can push the system step by step to its load limit, and beyond.
Finding similarities also helps with error messages, test specifications, and the log files of the test object, with generating test data from data format descriptions, and with deriving test sequences from code analysis. Large language models have opened up another field. They draft test cases from requirements, write test code, and explain error messages.
We still have to review what they produce before it goes into the test suite. CT-AI has learners practice exactly that: a language model generates test cases using boundary value analysis, and the learners check the output for correctness and completeness. Using generative AI for testing now has its own syllabus, the ISTQB Certified Tester Testing with Generative AI (CT-GenAI).
Where the Limits Lie
One especially tempting idea is AI as a test oracle, the source of the expected result. It immediately raises a follow-up question: if an AI is good enough to determine the expected result, couldn’t it be the system under test itself? Perhaps even a better one than the original?
CT-AI describes such proxy oracles, for example a second model that assesses the outputs of the first. It also urges caution, since the second system can bring similar biases and errors with it.
That leads to the question of limits. Which decisions can we, and do we want to, leave to an AI? Some people are reminded of the trolley problem, which is barely solvable even for humans: if a fatal accident is unavoidable and you can still influence the outcome, who gets to live and who has to die? No test case answers that. But the question belongs on the table before an AI makes that decision in the field.
Frequently Asked Questions
Why don’t impressive demonstrations of AI systems prove that they can reliably recognize objects?
Because the path to the result is often different than expected. Studies have shown that images were classified as pictures of horses not because of the horses depicted, but because of a patch of forest in the background or the signature of the photographer, who frequently photographed horses. This is reminiscent of the horse “Clever Hans,” who only appeared to count.
Does a fatal accident involving an autonomous vehicle argue against the use of AI in safety-critical areas?
Not necessarily. The 2021 article pointed out that in Brandenburg alone, an average of two to three people died each week in traffic accidents. Even imperfect AI can offer advantages in such situations. The technology is alternately hyped and condemned before the full picture is clear. A prerequisite for its use remains that it can be adequately safeguarded.
Why is a single metric insufficient to assess the quality of an AI classifier?
Because the significance of an error depends on the domain. The confusion matrix allows predictions and reality to be compared for binary classifiers, while the F1 score represents the harmonic mean of precision and recall. In medicine, diagnosing a tumor that does not exist is far less serious than overlooking one that actually exists.
Can the test levels of the V-model be applied to systems that incorporate AI components?
For complex systems that incorporate one or more AI-based algorithms, dividing them into test levels makes perfect sense. It remains unclear whether this also applies to machine learning with a large number of hidden layers. This question leads directly to the issue of the implementation’s explainability. It is also unclear whether white-box techniques are effective here at all.
What role does IT security play in testing AI-supported systems?
A significant one, because even minimal manipulations of the environment can skew the results. If an autonomous vehicle interprets the “30” on a speed limit sign as “80,” it will drive through the city at a correspondingly high speed. Similarly critical are inconsistent situations, such as a stop sign on the highway. Security aspects are therefore just as important as functional testing and non-functional testing.
What risks arise when an AI continuously learns during operation?
The system may become overly specialized. A commuter’s self-driving vehicle would soon be trained on the specifics of their daily route, while forgetting everything else. One alternative is to limit learning to service or development. The appropriate level of restriction depends on the application domain.
How can artificial intelligence itself be used in software testing?
One obvious area is performance testing: Depending on the input data, the AI detects anomalies in system behavior and system load and can gradually push the system to its load limit and beyond. Other applications arise from identifying similarities (for example, in error messages, test specifications and log files) as well as in the generation of test data and test sequences.
Can AI serve as a test oracle?
The idea is appealing, but it raises an uncomfortable question: If an AI is good enough to determine the expected result, it could potentially serve as the system under test itself, perhaps even better than the original. This also raises the fundamental question of which decisions we are willing to leave to an AI in the first place.
