Test design with AI

AI-supported test design refers to the use of language models such as Co-Pilot or ChatGPT to systematically generate equivalence partitions, boundary values and test cases. The systems deliver usable results in minutes, but make mistakes. If you want to evaluate the results, you need to master the classic test design methods yourself.

Key Takeaways

• Test design methods such as equivalence partitions and boundary value analysis are hardly ever used in practice, even though almost all testers have learned them in the certificate course.
• AI systems deliver test design results in 30 to 40 seconds, which, with post-prompting, reach a level of quality in two to three minutes that Michael Fischlein has rarely seen in customer use.
• Anyone who wants to evaluate AI-generated test results needs methodological knowledge themselves, otherwise they will lack the ability to recognize errors and gaps in the output.
• Boundary value analysis is the test design method with the broadest utility, because boundary value errors occur systematically and even an imprecise formulation such as “between” leads to incorrect test derivations.
• Hundreds of real test cases can be reduced to half by equivalence class analysis because values from the same class are redundant, while other areas remain completely untested.

Test design from the gut is the rule, not the exception

Many experienced testers work without formal test design methods, even though they have learned them. Test data, requirements and test cases arise from experience and gut feeling, not from equivalence partitions or boundary value analyses.

This is already evident in foundation-level training. Even people who have been testing for ten or twenty years and are only now getting their certificate are learning the basics anew. They may have intuitively understood the concept of equivalence partition, but they don’t know the name. They vaguely remember boundary values. The larger test design methods that come afterwards are largely unknown.

This is not necessarily a problem. Anyone who is satisfied with the quality of their deliveries and delivers on time is doing a lot of things right. This is often made possible by a person in the team with a keen sense for the software, a kind of real intelligence that finds errors without the need for formal technology.

Why test design methods do not find their way into practice

The main reason is a lack of practice. In the four-day Foundation Level course, only about half a day to a full day is spent practicing test design techniques. This is not enough to internalize them.

Real projects are more complex than the standard examples from the training course. A coffee machine, a Coke machine or an ATM are easy to understand. A decision table with five conditions results in 2 to the power of 5 lines, which is manageable. With twenty conditions, it becomes a huge table that many people shy away from.

Then there’s the blank sheet of paper. And we all know the fear of the blank sheet of paper.

Michael Fischlein

The agile world exacerbates the problem if it is misunderstood. “We document less” becomes “we don’t write anything down”. Testers believe they have to be fast and don’t take time for test design. Under time pressure, many people shy away from the investment of laboriously practicing a method first.

There is also often a lack of organizational commitment. Test managers with an advanced level certificate have the paperwork, but not necessarily the knowledge, to demand and exemplify test design in the team.

Exploratory testing does not replace methods, it needs them

The most common way out is exploratory testing: simply testing on the side, without formal design. But good exploratory testing requires you to master test design methods.

Anyone doing exploratory testing should implicitly apply equivalence partitions, boundary values and decision values. It is precisely this knowledge that helps when roaming through the software: which price category, which value ranges, which combinations are relevant. The methods do not disappear, they work in the background.

The simple methods provide the greatest leverage

Boundary values and equivalence partitions are the two techniques that are almost always used. They are simple and cover the majority of typical errors.

Boundary values regularly go wrong in practice because requirements are formulated textually. A sentence such as “the shortest board is between one meter and five meters” is ambiguous. Does “between” mean that the limits are excluded, i.e. 1.01 meters and 4.99 meters? This is obviously not what is meant, but this is precisely where errors arise.

Equivalence partitions drastically reduce the number of test cases. For one customer with tens of thousands of test cases, more than half could be deleted after analysis because they were duplicates from the point of view of the equivalence partitions: different values, same class. At the same time, entire value ranges were not covered at all.

A simple exercise question from recruitment interviews shows how poor the basic understanding is. If you are asked to test a closed interval from 9 to 99, you should choose one value below, one within and one above. Instead, some candidates wrote down one, two, three, four up to 100 on the flipchart without realizing how pointless this is.

State-based testing is also suitable for everyday use if the system has states. A ticketing system can be easily visualized and understood. The important thing is: no rocket science, but simple methods that you can apply hands-on, even under agile time pressure.

AI closes the method gap, but only half of it

Large language models deliver usable test designs in seconds, even though they are not trained for testing. Systems such as ChatGPT or Copilot evaluate language statistically and generate test cases, acceptance criteria, boundary values and equivalence partitions on demand.

The results are not flawless. Hallucinations and nonsense are included. As a rough estimate, the results are about 80 percent correct. If you have the same task solved by experts, the result is similar at 80 to 90 percent, but takes one to two days. The AI needs 30 to 40 seconds; with a little prompt optimization, a concept is ready in two or three minutes.

AI thus pragmatically bridges the gap between “method learned” and “method never used”. But a second, more difficult gap remains: someone has to evaluate the result. Anyone who has not mastered the methods will not recognize the errors in the AI output.

This is where the actual training task arises. An experienced trainer sees the error in the complex test design immediately. The junior consultant directly after the foundation level does not see it. Building up this assessment ability is the challenge ahead.

How to get started with AI-supported test design in practice

Talk to the systems like a person. They are chat systems, a “please” in the prompt doesn’t hurt. Formulate clearly what you want, for example: please write out the equivalence partitions for this state.

A feasible process starts with the technical context and works its way to the method:

• Set context: ‘I am an insurance company and want to create new customers. What variables and fields do I have, how are they attributed?”
• Refine variables: more fields, fewer fields, adjust until the list fits.
• Apply the method: “Build the equivalence partitions from this.”
• Sharpening: deepen individual areas, add missing points.

This also works in pairs, by bringing the AI to the table as a third person and discussing the results together. In training courses, the error in the output becomes a learning moment: anyone who recognizes that an AI answer is wrong has understood the method.

There is a limit to filling. Customer data and sensitive information do not belong unchecked in a chat system. A requirement for a common store system without a name in it, on the other hand, can be used via copy-paste to generate equivalence partitions and see the first gaps.

It is remarkable where the systems draw their domain knowledge from. In the case of a requirement from the automotive sector relating to reversing and safety systems, Copilot provided usable requirements including source links. The sources were the publicly available pages of the manufacturer itself. One week of planned work was almost done, minus the details that were still missing.

Playing is the fastest way to learn

The best way to get started is to try it out, not read about it. By playing, you learn what prompt engineering means in practice and how a system reacts to roles, facets and formulations.

The effect of a formulation only becomes tangible when you do it. If you ask the system to act in a certain role or introduce an additional facet, the response changes. Sometimes this results in nonsense, sometimes exactly the hit you need. You can only build up this feeling by stimulating it yourself.

The same principle can be applied to technical work. If you build a story or scenario with AI, you learn the same mechanics that will later be used when writing test cases and requirements.

Know-how remains the bottleneck, even if the basic work disappears

AI systems will take over the simple, repetitive test design work. What remains is the need to formulate a problem correctly and check the result using common sense.

This shifts the emphasis back to requirements engineering. You have to make it clear to the system what you want and then look at the result with expert knowledge. Both require built-up know-how.

A comparison makes the open question clear. Today, hardly anyone knows how a compiler works, yet everyone relies on it to do the right thing. It is conceivable that AI-supported testing will follow the same path. The unresolved question is how testers will build up the know-how to be able to evaluate the results at all in the future.