Test design methods

Test design techniques are structured methods that testers use to generate verifiable test coverage. They can be divided into four groups: process-oriented, condition-oriented, data-oriented and appearance-oriented. Anyone who masters at least one technique from each group covers around 90 percent of typical test situations in practice.

Key Takeaways

• There are between 25 and 30 test design techniques in total, but most testers only really need about ten of them in their day-to-day work.
• All test design techniques can be divided into four groups: process-oriented, state-oriented, data-oriented and appearance-oriented, and a tester should master at least one technique from each group.
• If you only know techniques from one group, you are falling into a trap: a state-oriented technique will not help if the actual problem is data-based.
• Developers often only test the happy path during unit testing and neglect error cases, although equivalence partitioning is enough to close this gap.
• Templates for techniques such as decision tables significantly lower the barrier to entry because learners no longer have to construct the table structure themselves, but fill it in directly.

Why testers often ignore test design techniques

Many testers know test design techniques but don’t apply them. Rik Marselis has been teaching these techniques for almost 25 years, and it is precisely this gap between knowledge and practice that frustrates him.

The first reason is banal: No one requires testers to use a technique. Where there is no expectation, the structured approach is the first thing to fall away.

The second reason depends on the quality of the system. If the software quality is poor, there is no need for sophisticated technology. You press a button and you have found the error. Only with good quality does a technique show its value, because it proves what coverage has actually been achieved.

The third reason is the hurdle of proper learning. Anyone who passes a certification exam knows the basics and can tick answer A, B, C or D. The real application is more complicated than any exam question, and this is where many drop out.

Too many techniques are themselves part of the problem

Those who go through ISTQB Foundation, Advanced Test Analyst and Technical Test Analyst learn around 15 to 20 test design techniques. There are around 25 to 30 in total, depending on whether you count variants as a separate technique.

This quantity is overwhelming. Previous TMAP certifications required knowledge of 19 techniques and approaches, which simply could not be taught in a three-day course. The result was an exam that you passed without really having learned anything.

TMAP has drawn a conclusion from this. The entry-level course teaches five techniques and one approach, with enough time and practice to actually use them at beginner level. Fewer techniques, but applicable.

Four groups organize the entire field

All test design techniques can be categorized into four groups. This categorization, developed by Rik and his colleague Bert, makes the field manageable.

• Process-oriented: Path testing with different levels of path coverage, state transition testing.
• State-oriented (condition-oriented): Decision tables where conditions define the test cases.
• Data-oriented: Equivalence partitioning where a data element has multiple partitions.
• **Appearance: syntactic checks such as buttons, fonts, colors and non-functional aspects such as performance and usability.

The practical consequence is clear: master at least one or two techniques per group. If you only know techniques from a single group, you are falling into a trap.

One team that Rik accompanied as a coach opted for the Simple Comparison Test as the standard because it promised high efficiency and effectiveness. In practice, it did not work. The team had a data-related problem, and a condition-based technique is the wrong choice for this.

Which techniques form the basic set

The starter course covers all four groups with five techniques and one approach. Specifically, these are equivalence partitioning and boundary value analysis from the data group, path testing from the process group, the decision table from the condition group and syntactic testing from the appearance group. In addition, there is exploratory testing as an experience-based approach.

A second course on high-performance quality supplements one further technique per group. Together, this results in around ten techniques and two approaches.

This toolbox is sufficient for the majority of the work. Around 90 percent of testers need no more than this over their entire career.

How to choose the right technique

Four criteria decide which test design technique is right for you.

1. The type of test problem: Which of the four groups do you choose from?
2. The quality characteristic: If it’s about performance, for example, you’ll quickly end up in the Appearance group.
3. The risk: High risk requires more coverage.
4. The skills of those involved: If the team doesn’t know a technique, you either choose a known one or you train people.

Three levels of risk are sufficient: low, medium, high. Percentages are misleading. If you claim 72 percent, you have to explain why not 69. Since the techniques usually only offer three coverage options anyway, three risk levels fit in perfectly.

Templates lower the barrier to entry

Templates make techniques learnable that were previously considered difficult. One TMAP trainer admitted that decision tables were difficult to teach for years. A freely available template changed that.

The reason is simple. With a template, you no longer have to explain how the table is created. You show the finished structure and the learners fill it in: Set conditions, define actions, determine the expected result via true-false combinations. The expected result is the core of every test case.

Not every technique needs a template. Marginal value analysis is so intuitive that people often use it without ever having heard the term.

Path test is the underestimated technique

Path testing covers all paths in a business process and is particularly suitable for acceptance testing. ISTQB is the only process-oriented method that teaches state transition testing, which requires states. However, not many systems have states.

In acceptance testing, you want to test business processes, not states. This is exactly what the path test, also known as the process cycle test, is made for.

In a current project, business analysts are carrying out the functional acceptance testing. Because they had good process flows, Rik suggested the path test to them. After an hour and a half of explanation with practice and a two-hour session on a real process flow, the first test cases were ready, created step by step using a template.

Coverage and experience go together

A technique alone is not enough. It proves that you have covered everything that is important for your risk level. However, confidence only comes from the combination of cover and experience.

In that project, the analysts found a situation that their test cases did not cover, even though all paths were covered. Rik’s advice: Add a test case based on your experience. But don’t throw one away, because you need the covered cases for the proof.

This is where coverage-based testing meets practice-oriented testing. Add a test case based on your experience, but don’t throw one away because you need it for your coverage.

Rik Marselis

The higher the risk, the more thorough the test planning. Experience-based techniques such as exploratory testing should always be combined with planned techniques.

Developers also benefit from simple techniques

When it comes to unit testing, code coverage often counts for developers, but “100 percent coverage” says little as long as it remains unclear what kind is meant.

Line coverage is the weakest statement. Some developers pack five statements into one line, and one executed statement already generates 100 percent coverage of this line. Instruction coverage is better, so that every instruction has been checked.

The most valuable is decision coverage, because decisions are the most important instructions in the code. A move statement has a high probability of working. With if statements and while loops, however, you want to know whether the logic meets the good case and the error case.

This is where equivalence partitioning, the simplest data-oriented technique, helps. Many developers only test the happy path and forget the error case. This technique provides a test case for each class, the right one and the wrong one.

The real lever is awareness, not teaching. If you tell developers that they should also test what happens if they enter the wrong thing, it makes sense. The reaction: Then I have twice as many test cases. That’s right, you do.

The technology stays, the application moves into new hands

There are no new test design techniques in sight. All known techniques are in books that are over 20 years old.

The earliest is “The Art of Software Testing” by Glenford Myers, around 45 years old, with boundary value analysis and equivalence partitioning. There is also the work by Boris Beizer and the old TMAP books. ISTQB often refers to Beizer and partly to Myers.

What changes is not the canon, but who uses the techniques. When business analysts apply the path test to their business processes, this corresponds exactly to their view of the system. The technology is old, the user is new.

Three techniques that work in practice

From Rik’s point of view, three techniques stand out.

The Path test is at the top because it covers business processes completely and works in practice. The decision table checks every single possibility reliably, but gets out of hand if there are more than five conditions because it gets too big. In most cases, far fewer conditions are sufficient, so it works well.

The Simple Comparison Test is the most complex of the three and provides the most comprehensive coverage. It tests multiple decision points, each with modified condition-decision coverage. From these test situations, test cases are combined from the beginning to the end of a process.

Its advantage is evident where the path test does not cover enough. If different combinations of outputs lead to the same paths, the path test only provides one test case, while the simple comparison test provides all possibilities. This results in a few more test cases, but the highest achievable coverage. If a system passes these test cases, you can be confident that it works.

However, a full-day workshop is required to learn this technique properly. The right template exists.