Test design with model-based testing

Model-based testing refers to an approach in which test cases are systematically derived from a formal model of system behavior. The central problem in practice is the gap between the abstract model and concrete test execution. A two-phase approach bridges this gap: First, low-level test cases are created from the model, then concrete test data and automation scripts are added interactively.

Key Takeaways

• Model-based testing often fails in practice due to the gap between abstract test design and concrete test implementation, because concrete test data and GUI actions make the model unnecessarily complex.
• A two-phase approach solves this problem: first, high-level test cases are generated from the model and kept manually executable; only then are concrete data and automation scripts added.
• According to Capgemini’s World Quality Report, test automation has not yet achieved the set business goals: neither smooth CI/CD pipelines nor efficiency increases nor measurably better software quality.
• Systematic test design procedures such as equivalence partitioning or boundary value analysis are hardly used by trained testers in practice; instead, gut feeling dominates when writing test cases.

What model-based testing is and where it gets stuck

Model-based testing derives test cases from a model instead of writing them individually by hand. The tester defines what is to be covered, models it and generates the test cases from it. The idea is old and attractive. In practice, it rarely runs smoothly.

Matthias Hamburg has introduced the approach as a test manager in several projects. His verdict is sober: “It never went so smoothly.” This is in line with industry observations. The World Quality Report by Capgemini and Sogeti states that, overall, test automation has not achieved the business goals associated with it.

Three expectations remained unfulfilled: smooth CI/CD operations with automation, the hoped-for increase in efficiency and measurably better software quality. Managers and company management were interviewed who were looking at business objectives. Exactly these goals were missed.

Why most MBT approaches fail in reality

Two hurdles stand in the way of model-based testing. The first concerns the people, the second the tool workflow.

The first hurdle: Those who create models are usually not trained in modeling. Formal models require training that testers rarely have. Therefore, only models that remain understandable without a long lead time and are close to the tester’s way of thinking are practical.

The second hurdle lies in the process itself. Model-based testing does not currently offer a seamless transition from modeling to the actual test. It is precisely in this gap that the problems arise.

Where the gap is in the test process

The ISTQB model of test activities clearly organizes the steps: test analysis, test design, test implementation, test execution. The test design defines what is to be covered, i.e. the test conditions. Test cases can then be generated from the model. Execution can be automated using tools such as Selenium or Playwright.

In between lies test implementation, and this is the crux of the matter. This is where concrete test data is selected, where it is determined which buttons are pressed and which concrete results are checked. It is the step from the abstract to the concrete.

The usual response of MBT tools is to pack the concrete data into the model. However, this makes the model very complex. A model should abstract, provide a logical view. Including concrete data breaks this view. This overloading is precisely one reason why model-based testing has not become established.

The two-phase approach closes the gap without overloading the model

A more recent approach separates abstraction and concretization into two phases. First, high-level test cases are created from the model. These must be manually executable, because in manual testing, people fill in the gaps themselves anyway.

Phase one can be completed early, before the software is finished. This is the shift left advantage: modeling early ensures the quality of the test basis and the specification. Then you stop without fear of a paradigm shift.

Phase two begins as soon as the real application is running. The tool executes the high-level test cases and asks back at the open points: What does “enter text” mean in concrete terms here? It displays the dialog interface, you enter the appropriate text and have the action recorded.

This is not a simple capture-and-replay. Earlier capture tools generated scripts that often no longer worked afterwards, with corresponding maintainability problems. Here, the action is first generated at GUI level from the functional level. This action is readable and can be re-edited in a low-level test case. The executable script for the automation platform is then created from this, in the tested case Playwright.

Why this approach remains maintainable

Maintainability is the real benefit. If an action changes because the implementation has changed, all places where this action occurs are updated. The redundancy is not formally reduced, but the tool keeps it consistent.

A typical practical problem demonstrates the benefits particularly clearly. Some UI frameworks generate new IDs for the GUI elements with every small change. As soon as a button is added somewhere, everything behind it is renamed. This breaks classic automation.

The tool handles this pragmatically. It stops at the point where it cannot find a selector and asks for it. You adjust the selector in the list and then the test runs again. Only what has actually changed needs to be updated.

Action State Testing: a state model in the language of testers

The tried and tested tool comes with its own model, coupled to a black box procedure called Action State Testing. In terms of content, it is close to state-based testing, but is formulated closer to the way testers work.

The model is currently text-based, not graphical. You write down test steps in the same way as you are used to when testing: an action, an expected result to be tested, a subsequent state. The subsequent state of a step is the prerequisite for the next one.

An example from the glossary application makes this clear. Initial state: the search is displayed. Action: enter a search text. Check: the number of hits is at least one and the search text is included in the hits. Subsequent state: the results list is displayed. If you select an element, the details appear and a back button takes you back to the hit list.

The sequence of steps can be displayed as a flow chart. The result is scenario-based and also uses the states, a test-relevant state model that can be systematically covered.

One procedure alone is not enough

Model-based testing does not replace the classic design methods, it combines them. In addition to the state-based approach, equivalence partitioning was also used in the glossary app. We searched not only for texts from the name of a term, but also for abbreviations and synonyms, because the search had to work for all three.

This is a common weakness in practice. Many testers learn equivalence partitions, boundary value analysis, state and decision tables once, but tend to apply them on a gut feeling basis in everyday life. A structured approach, combined with modeling, brings these procedures back into daily work.

A negative example from industry shows just how far things can go without such discipline. A large company with quarterly releases had all the specialists come into the office on the release weekend and test on an ad hoc basis. When asked when any bugs found would be fixed, the answer was: “We hope there aren’t any.” A clear sign that trained testers were not doing their job and did not know any test techniques.

Open construction sites of the approach

No tool is ready, and this approach is new to the market. Two wishes remain.

• Data-driven testing: Test data such as the search text should be stored separately and reused in further test cases instead of just being a loose text reference in the step.
• More design procedures: In addition to state-based testing, equivalence partitioning and boundary value analysis directly in the tool would be helpful in order to achieve the desired test coverage cleanly.

Despite these gaps, the application could be tested extensively. Even if the glossary app is not commercially critical and nobody loses a cent, it remains a question of attitude: a testing expert will not hand over an application that has not been well tested.

How to get started with model-based testing

People often shy away from model-based testing because it is unclear what they can actually do with it. The pragmatic way to get started is through trial and error, not theory.

Download a trial installation of a good tool, there are several. Try out how it works and see what suits you and what you can handle easily.

If I have to study for days before I can even generate the first small test case, then don’t bother.

• Matthias Hamburg

Accessibility is the benchmark. One advantage of the tried and tested tool is the no-code generation. As a specialist tester, you specify directly in the GUI what is to be tested and how it is to be entered, without having to write an automation script. In previous large-scale projects, this required a separate sub-project consisting of automation specialists who implemented what the specialist testers had designed.