The system testing is a very exciting test level. unit testing and integration testing focus more on the inner aspects of the application. In system testing it is much more about the external view. For the first time, the software is viewed as a black box and tested against the business requirements. These now also play a much greater role than in the lower test levels. This means that technical employees, later users or customers can also actively participate in testing. However, this change in perspective also brings with it completely new challenges. For this reason, I have dedicated an entire book to this topic: "The system testing - From Requirements to Proof of Quality".
Definition system testing
The ISTQB defines system testing as "a test level focused on verifying that a system as a whole meets specified requirements."
The big difference to previous test levels is that here the integrated overall system is tested against the requirements. The system is viewed from the outside like a black box.
Test base
Functional and non-functional requirements usually serve as the test basis for the system testing . However, user stories, business processes or user documentation are also suitable. If this basis is missing or incomplete, users or specialist departments can be questioned. The old system can also serve as a test oracle, e.g., for system replacements or migrations.
Test case creation for system tests
Functional requirements are often available in text form. How can test cases be created from these requirements? In practice, a combination of two approaches has proven successful:
- Structured test case generation: Systematic methods such as equivalence class formation, limit value analysis, decision tables, etc. can be used to analyze requirements in a structured manner and derive test cases from them. This ensures that the necessary breadth and test case coverage of the requirements is given.
- Experience-based test case generation: Requirements documents can never completely represent the system in its entirety. There is too much transfer loss from the idea to the written requirement. This means that it is always advisable to define test cases on the basis of your own experience and also the expectations of the users.
This interaction brings a good set of test cases for the system testing. In practice, it is often the case that many questions arise when creating test cases for the system testing . If I want to create a concrete test case, the sentences in the documents must also be concrete. And this is often not the case: "The system must be performant", "The application should be easy to use", "The button should be green". Gaps also quickly become apparent here. All these queries need to be clarified.
The clarified points must of course also be known to the developer. It is therefore all the more important to start creating test cases at an early stage.
Test object
The test object for the system testing is the integrated overall system. I.e. the software or application is tested via the user interface or the interfaces. For a final evaluation, the system must therefore also be complete.
Integration with other systems is not part of system testing, but is often referred to as system integration testing.
Test objectives of the system test
The goal of system testing is to check whether the functional and non-functional requirements for the application have been met and sufficiently implemented. In practice, some challenges lurk here that must be considered at an early stage (see also typical problems).
Test environment for system tests
This is also where a difficulty lurks. For a valid statement about system testing , the test environment must correspond to the production environment. Or at least very similar to it. Especially when it comes to non-functional test types such as performance testing or reliability. And this duplicate infrastructure can be expensive. This is also where a massive advantage of virtual machines and cloud solutions becomes apparent. If the later productive system is a parameterized virtual machine, it can also be cloned cost-effectively for the system testing .
Test data
Test data management becomes much more demanding from the system test level onwards. Unit and integration tests usually still handle test data in the local environment of the test case. At system testing , however, much more extensive test data sets may be required, e.g., created contracts, historical data, linked data sets, etc. Two procedures have become established in practice:
- Real data: If available, data extracts can be used from productive systems, for example, to use as test data for the system testing . This is particularly popular during the maintenance phase. Where necessary, this data is anonymized. But always with the awareness that this also changes the data. The advantage of real data is, of course, its practicality, but also the fact that sometimes data and constellations occur that are not covered by requirements and test cases.
- Synthetic test data: Real data is far from covering all possibilities, corner cases and combinations. Therefore, it is also advisable to generate concrete synthetic data to test special cases: Birthday on 2/29 or postal codes starting with 0 are classics here.
Test automation of the system test
For a long time, test automation at system testing was poor. Tests under the hood, such as unit and integration tests, were well supplied with tools simply because of the proximity to development. However, with the advent of agile projects in particular, there has been a massive push here at the system test level. Both in test automation solutions for surface tests, web tests and also interface tests. There is a wealth of possibilities here today. I have summarized how to implement test automation here in concrete terms in the book "Basiswissen Testautomatisierung".
system testing in agile projects
The model of test levels comes from a time long before agile projects. It is therefore often ignored in Scrum and the like. But if we take a closer look at the model, we see how important the ideas and aspects of the test levels are in agile contexts as well. Like at system testing: Test the system from the outside. Prepare test data. Consider test cases. And of course set up test automation.
In this context, the model of the test pyramid appears again and again. It provides a similar perspective to the test levels.
Therefore, it is also worth looking through the system testing glasses in agile software development. The intention of the system test can be transferred and the relevant aspects can be used in one's own project.
Typical problems
When I look at the projects of the last few years, the same challenges in everyday life occur again and again at system testing :
- Non-functional requirements are not considered. system testing includes more than just functional requirements. Non-functional requirements must also be considered here, such as usability tests, performance or reliability tests.
- The system testing is the first test level, which is completely based on the requirements. And from requirements to code and test cases, there is inevitably a media break. Natural language becomes technology. While development can creatively fill the gaps, a tester cannot do so to create test cases. Here, the same target image must be worked on again and again.
system testing in practice
In practice, the system testing is often the test level to start with. This is because this test level is easier for departments, testers and management to grasp. Take the requirements and test the system against them. This is easier to understand than arguing any small-scale unit tests at code level.
In this situation, however, the system testing often shows the gaps of the other test levels. Instabilities and inconsistent behavior of the software then indicate that there is a lack of robustness in the substructure. Due to the late testability at system testing , these findings unfortunately come very late.
Therefore, it is particularly important to establish the other test levels as well. Only one system testing is too little.
Even if the execution of the system test is possible quite late, the test cases, for example, can already be designed before the first line of code is written. Questions and ambiguities are thus resolved at an early stage.