What Is Automated Test Design?
Test automation exists to make any test activity more efficient, and that starts before a single test runs. Automated test design asks how much of the design work a tool can carry and in what form the test cases exist. There are five levels, each with its own approach:
- Capture & replay
- Script-based
- Data-driven
- Keyword-driven
- Model-based
With each level, the demands on test design go up. Techniques like capture & replay deliver results faster, while keyword-driven and model-based techniques give you solutions that last and scale.
Which test cases a test object needs is a different question, answered by test design techniques such as decision table testing or state transition testing. The five levels come in after that. They decide what shape a test case takes and how much of the work a tool does.
Why look so closely at test design when the tests are run automatically anyway? Because software keeps changing, sometimes more often, sometimes less. Requirements change, bugs get found and fixed. When that happens often, the manual effort of maintaining automated test scripts can easily wipe out the benefit of running them automatically.
Capture & Replay: Quick to Record, Expensive to Maintain
Capture & replay is the entry level: very cheap to start with and very expensive to keep running, because every change to the system means recording again.
Capture & replay means recording the manual execution of a test case (capture) and then repeating that recording as often as you like (replay). This needs a test tool that can record what the tester does and later reproduce those actions at the interface of the test object. The demands on the tool grow with the complexity of the test object. In extreme cases, that can mean test robots that add, change or remove hardware components. In the simplest and most common case, the technique is used to test through a web browser: the tool operates the browser and observes what happens. For the recording, it only has to store the mouse and keyboard actions on the user interface, along with the position and content of the expected responses.
There are two ways to do this: store the absolute coordinates of every mouse click, or store the object IDs of the clicked elements, if they have IDs. The second option has the advantage that rearranging elements on the screen doesn’t immediately force you to record the test cases again. One of the best-known tools for capture & replay testing of web applications is Selenium. Its browser extension, Selenium IDE, records in Firefox or Chrome, and the recordings play back on many of the most widely used browsers.
Selenium is a very powerful automation tool. It also supports the higher design levels, and in practice that is how it’s usually used.
The big advantage of capture & replay is that domain experts can create and run test cases themselves, without help from test or software specialists. There is hardly any preparation or learning curve. The main drawback is the comparatively high maintenance effort. Every time the system under test changes, the test case has to be recorded again: when the content changes, when the order of steps changes, or even with tiny changes to the page layout (for the variant based on absolute coordinates). Even fairly small changes mean re-creating most of the test cases.
Because creating a test case means recording a manually executed test, every test case has to be run manually again. A single change takes low to medium effort, depending on how complex the system is. But since the test object changes so often, the adaptations pile up and test design costs end up very high. So when is capture & replay the better choice over script-based test automation? Mainly when you want to show results quickly, when changes to the system are the exception rather than the rule, or as a complement to the approaches described below.
What Does Script-Based Test Automation Add?
One level up, the initial costs stay low and the recurring costs drop to medium. You can change test cases without running them again, but someone on the team has to be able to program.
In script-based test automation, test cases are designed as executable scripts, in other words as executable test code. The test designer needs programming skills and a development environment. Test scripts can be written in any programming language. At the component test level, xUnit tests are common: unit tests written in the same language x as the program under test. That means JUnit for Java programs, CppUnit for C++ programs and so on. At the system test level, the choice of language depends mostly on the test interface of the system under test.
The main advantage of script-based test design over capture & replay is that the definition of a test case is independent of its execution. You can change the test design without running the test again, which pays off especially for larger tests. The main drawbacks, however, are similar to those of capture & replay: fairly often, a change means adapting test cases by hand. It happens less frequently, though, because programming techniques make it relatively easy to absorb changes such as moved screen elements into the test design. One way is to move the positions of the screen elements a test queries into separate classes.
The key benefit is that a necessary change doesn’t automatically mean running the whole test case again. You can weigh whether adapting the test code without running the test takes more effort than executing the test case manually and logging it automatically, as in capture & replay. That depends on the situation. Our experience, though, is that for longer or time-critical tests, manual execution costs far more than adapting the script. There can be other factors as well, such as screenshots used in test cases or data that is laborious to create during test execution. Both make adapting the script much easier than running the test.
The error-proneness of manual test execution shouldn’t be underestimated either. Every mistake is logged automatically, and a wrong click can also change the state of the system unintentionally. The recording would then have to start again from the beginning, which drives up the effort for capture & replay even further. Overall, the advantages of the script-based approach outweigh those of capture & replay.
How Does Data-Driven Test Automation Work?
The test logic stays the same, and each new row of values is another test case.
Unlike the two previous approaches, data-driven testing puts the data at the center. It starts from a test design that abstracts from concrete data, for example by specifying only the expected parameters but not their values. The tester’s job is then to define suitable data. The data can be kept in any format that fits the framework in use. In practice, Excel sheets or SQL databases are common. One way to set up the test design is a table with one column for each required input parameter and one for each expected output parameter. The tester fills the rows with concrete values. An adapter connects the table to the test object, passes in the input parameters and compares the actual results with the expected ones.
A typical tool for this is FitNesse. FitNesse is a wiki web server you can install locally on your PC, so you can create the data directly as a table on a dedicated test wiki page. Alternatively, in edit mode you can import the values from an Excel sheet with the “Spreadsheet to FitNesse” button. Clicking the “Test” button starts the test run. FitNesse then colors each row green or red, depending on whether the actual result matches the expected one.
Keyword-Driven Test Automation: More Setup, Less Maintenance
Here you invest more up front, in a shared language of keywords, and pay less later, because many changes only have to be made in one place.
The two approaches described so far rely on programming languages and focus either on behavior or on data. The next two aim at better maintainability through additional layers of abstraction.
In keyword-driven test design, test cases use abstract keywords instead of concrete commands. At runtime, an adaptation layer replaces the keywords with concrete commands, depending on its configuration. Changes in system behavior that don’t require new vocabulary can then be carried over into the affected test cases with relatively little effort, by swapping keywords. Many teams see the improved readability as the main benefit. New scenarios still need new test cases, and new system actions or responses mean adding keywords. Plenty of tools support keyword-driven testing, and any programmer can build a simple framework of their own. More powerful tools with editor support, for example auto-completion, are Xtext and Cucumber with its official IDE extension. In Xtext, a grammar defines the entire permitted language; in Cucumber, the step definitions do.
The main advantage of this approach is the higher level of abstraction, which makes test cases comparatively easy to create and simpler to maintain. In many cases, no major adaptation is needed at all, because the adaptation layer turns the test cases into concrete commands anyway. Many changes, for instance to the concrete interface format, can be made once in the adaptation layer, with little effort, for all test cases at the same time. On the downside, the preparation is moderately complex: the language for the keywords and the tasks of the adaptation layer both have to be agreed with all stakeholders.
From a purely technical point of view, you could get the same results with the script-based approach. But that takes a lot of preparation, because you have to build the abstraction layers yourself, such as wrapper classes or libraries that translate keywords into sequences of actions. It also takes discipline from the testers not to drop down from the higher level of abstraction whenever they need a quick fix, and leave behind test scripts that are hard to maintain. Modeling frameworks, by contrast, force testers to stay at the modeling level, which keeps maintainability high over the long run. And frameworks such as Xtext can generate convenient editors from the language description alone.
Model-Based Test Automation Lets a Generator Write the Test Cases
At the top level, building the model is expensive. After that, recurring costs are very low, because a test generator produces the test cases.
Model-based test design uses models as the basis for designing tests. Models are abstract descriptions of any kind of artifact, in our case artifacts used for test design. They can define system behavior or test behavior. They can be a separate test model or part of a shared test and development model. They can be deterministic or non-deterministic, timeless or time-dependent, discrete, continuous or a mix. They can describe data flow, control flow or both. The behavior can be continuous or event-driven. As a result, there is a wide range of ways to use all sorts of models for test design.
Models can describe not just individual test cases at an abstract level but entire sets of test cases. That works because branches, conditions, reactions to external events and much more are already defined at the model level. A model also gives you an overall view of the whole system and the dependencies between its components. That helps much earlier in development, for example when reviewing requirements or architecture designs. And of course, the model gathers all test-relevant information in one place, and the test cases are derived from it as a whole.
If the test cases are derived from the model automatically, you get further benefits. Small changes to the model that affect many test cases flow into those test cases automatically. The test generator often uses quality goals, such as reaching a certain coverage at the model level, as the exit criterion for test design. So even changes that add test steps, change the number of parameters or alter the expected behavior can be picked up automatically. The affected test cases are simply generated again.
Imagine you want to test an application that models possible career paths, where each step depends on passing an exam. There is one straightforward test case: the happy path, in which every exam is passed on the first try. Once you include the unhappy paths, even this small example shows that loops make an infinite number of paths possible in theory. A test generator usually follows only the structure of the model. When states are added, changed or removed, it includes them automatically the next time it generates the tests.
The main advantage over the keyword-driven approach is that you can now describe the entire flow, including branches, nesting, loops and parallel behavior, and use it for test design. A model-based test generator interprets the model and produces individual control-flow sequences from it, which are then converted into a predefined target format. Common target formats are executable test code, a human-readable version of it (for validation, for example) and test documentation. That means you can react to any kind of change at the model level or in the exports of the test generator, and the resulting test cases are generated automatically. The high risk of adapting large numbers of test cases by hand, with all the effort that comes with it in the approaches above, disappears. Test efficiency goes up considerably. The downside is the high preparation effort.
Automated Test Design Compared: Costs of the Five Levels
A qualitative comparison based on our experience shows the following cost structure:
| Capture & Replay | Script-based | Data-driven | Keyword-driven | Model-based | |
|---|---|---|---|---|---|
| Initial costs | very low | low | low | medium | high |
| Recurring costs | very high | medium | medium | low | very low |
The table makes it clear: higher levels of test automation save money in the long run, while lower levels are better suited to a quick assessment. The surrounding conditions always play a part in the choice, too. Which project approach are you using? Who writes the test cases? How much capacity do you have? You’ll find more guidance on choosing the right strategy in our German-language book “Basiswissen Testautomatisierung.”
The ISTQB Test Automation Engineering syllabus (version 2.0) paints a similar picture. It describes capture/playback, linear and structured scripting, data-driven testing, and keyword-driven testing as approaches for automating test cases, and it considers capture/playback feasible only for a small scope and a system that rarely changes. Model-based tools show up under test generation, the automated design of test cases from a test model.
Frequently Asked Questions
What are the types of test automation?
Seen from test design, there are five levels: capture & replay, script-based, data-driven, keyword-driven, and model-based. They trade setup effort against maintenance. Capture & replay is very cheap to start and very expensive to keep running; model-based test automation is the reverse. Script-based and data-driven sit in between, with low initial and medium recurring costs.
What are the most popular tools for automation testing?
For capture & replay of web applications, Selenium is one of the best-known tools, with the Selenium IDE extension recording in Firefox or Chrome. Script-based component tests often run on xUnit frameworks such as JUnit or CppUnit. FitNesse supports data-driven tests through wiki tables, and Xtext or Cucumber help with keyword-driven tests. Selenium also covers the higher levels.
What matters when designing an automated test system?
The conditions of the project come first: the development approach, who writes the test cases, and how much capacity the team has. Next comes how often the system changes. Rare changes can make capture & replay enough, while frequent ones justify the extra preparation for keyword-driven or model-based levels. From the script-based level on, the team needs programming skills.
