Pairwise Testing: Combinatorial Testing in Practice

What Is Pairwise Testing?

Pairwise testing (the most important form of combinatorial testing) is a test technique that selects, out of all possible value combinations of several parameters, a small test set in which every pairing of two parameter values occurs at least once. Instead of working through the complete Cartesian product, the interplay of every two parameters is checked, and guaranteed so for every pair. The tools optimize hard, but they do not reliably hit the theoretical minimum; finding it is computationally hard. What matters is the guarantee, not the last row saved.

Why this holds up has been studied empirically, though the picture is less smooth than it is often presented. In 2004, Kuhn, Wallace and Gallo analysed the failure databases of nine real systems for NIST and NASA, from a medical embedded device via web browsers and an HTTP server to a distributed scientific database. The shared finding: almost all failures hang on a small number of interacting parameters.

The numbers behind that finding, however, spread widely. Single parameter values triggered between 29 and 82 percent of the failures, depending on the system. Add pair interactions and you land between 47 and 97 percent, with three-way interactions between 89 and 99 percent. The study reports no fixed curve anywhere, and the authors state explicitly that a generalization would require considerably more data sets.

For practice, a clear conclusion remains: the jump from single values to pairs delivers by far the largest gain in every system studied, the step to three-way interactions considerably less. Where your own numbers land depends on the system. That pairwise is the most economical entry point holds across all nine.

Combinatorial testing counts among the data-based test design techniques. Beyond the pair it generalises: with t-way coverage, every tuple of t parameter values occurs at least once; pairwise is the case t equals 2.

When Is the Technique a Good Fit?

Wherever several parameters with clearly defined values interact and exhaustive testing would be uneconomical. Typical fields: configuration testing across browser, operating system, resolution and language; business rules with several factors; interfaces with many optional parameters; systematic authorisation testing.

The precondition is that the input space can be described as a finite list of parameters with finitely many values. Continuous value ranges are first reduced to a few representatives via equivalence partitioning. The two techniques interlock: equivalence partitioning goes deep into individual partitions and their boundaries, combinatorial testing goes broad across several parameters.

The savings grow with the number of parameters. With two parameters there is nothing to gain; the full combination is small anyway. From three parameters with several values each, the difference becomes noticeable; from four parameters, a factor of five to ten often lies between the full combination and the pairwise suite.

The Procedure in Five Steps

1. Identify the parameters. Which input parameters influence the behaviour together?
2. Determine the values per parameter. Reduce continuous ranges beforehand via equivalence partitions.
3. Choose the interaction level. Pairwise (t equals 2) is the norm; with a high risk level or known three-way dependencies, t equals 3.
4. Generate the test set. A tool computes the optimised test set that covers all pairs.
5. Execute and evaluate. If failures cluster around a particular parameter pair, that points to an interaction defect.

A Compact Example

The booking flow of a car rental portal has three parameters with three values each: vehicle class (compact, estate, SUV), payment method (credit card, PayPal, invoice) and pickup station (airport, railway station, city centre). The full combination would yield 3 × 3 × 3 = 27 test cases. A pairwise-optimised suite manages with nine:

Counting is worth it: every vehicle class meets every payment method exactly once, every payment method every station, every class every station. All 27 value pairs are covered, with a third of the full combination.

Suppose TC 6 fails in the test run: for payment by invoice with pickup at the airport, the credit check that invoice payment requires never runs, because the airport station uses its own, older checkout path. That is an interaction of exactly two values (invoice and airport), independent of the vehicle class. The pairwise suite finds such pair defects not by luck but by construction, because the pair occurs in at least one test case.

Coverage and Success Criteria

Two criteria complement each other in practice. Base choice coverage starts with a base test case built from the most representative values and then varies, per parameter, each further value one at a time; in the car rental example that yields 1 + (2 + 2 + 2) = 7 test cases. It ensures that every value occurs once, but says nothing about interactions. Pairwise coverage, by contrast, measures the share of covered value pairs out of all possible ones; the target is 100 percent.

What the levels mean in concrete terms, the example shows: three test cases are enough to show all nine individual values once, but they cover at most nine of the 27 pairs, a third. The pairwise suite with nine test cases reaches all 27 pairs and, along the way, nine of the 27 triple combinations. Whoever wants all triple combinations ends up, with three parameters, at the full combination, because 3-way and full combination coincide here. A proven rule of thumb: pairwise as the standard, plus targeted extra tests for individual combinations with particular risk.

Strengths and Limits

The strength is plain to see: guaranteed pair coverage at a fraction of the effort, well suited to automation, and the parameter model can be reused across releases. When a fourth vehicle class arrives, the model gains one value and the suite is regenerated, instead of dragging test cases along by hand.

Three limits come with it. First, pairwise finds no defects that arise exclusively from the interplay of three or more parameters; known three-way dependencies need t equals 3 or targeted additions. Second, constraints, meaning inadmissible value combinations (say: no SUV pickup at the railway station), demand tool support, otherwise the suite contains tests that cannot be executed. Third, pair coverage is only as good as the chosen representatives; badly cut equivalence partitions stay bad, however cleverly you combine them.

Tools take over the computation: ACTS (free, maintained by NIST), PICT (free, from Microsoft) or Hexawise (commercial). Constructing pairwise suites by hand is nowadays only interesting as a finger exercise.

Related Techniques

Equivalence partitioning does the groundwork: it forms the partitions per parameter whose representatives combinatorial testing then connects. When the combinations trigger clear business rules, decision table testing complements the picture, because it makes every rule explicit. The page test design techniques gives an overview of all eleven techniques.