The system testing is a very exciting test level. unit testing and integration testing focus more on the inner aspects of the application. In system testing it is much more about the external view. For the first time, the software is viewed as a black box and tested against the business requirements. These now also play a much greater role than in the lower test levels. This also leads to the fact that technical employees, later users or customers can already actively participate in testing. However, this change in perspective also brings with it completely new challenges. This was also the reason why I dedicated an entire book to this topic: "The system testing - From Requirements to Quality Proof".
Definition system testing
The ISTQB defines system testing as: "A test level focused on verifying that a system as a whole meets specified requirements."
The big difference to previous test levels is that here the integrated overall system is tested against the requirements. The system is viewed from the outside like a black box.
Functional and non-functional requirements usually serve as the test basis for the system testing . User stories, business processes or user documentation are also suitable. If this basis is missing or incomplete, users or specialist departments can be questioned. The old system can also serve as a test oracle, e.g. in the case of system replacements or migrations.
Test case creation for system tests
Functional requirements are often available in text form. How can test cases be created from these requirements? In practice, a combination of two approaches has proven successful:
- Structured test case creation: Systematic methods such as equivalence class formation, limit value analysis, decision tables, etc. can be used to analyze requirements in a structured manner and derive test cases from them. This ensures that the necessary breadth and test case coverage of the requirements is given.
- Experience-based test case creation: Requirements documents can never completely represent the system in its entirety. There is too much transfer loss from the idea to the written requirement. This means that it is always advisable to define test cases on the basis of one's own experience and the expectations of the users.
This interaction brings a good set of test cases for the system testing. In practice, it is often the case that many questions arise when creating test cases for the system testing . If I want to create a concrete test case, the sentences in the documents must also be concrete. And this is often not the case: "The system must be performant", "The application should be easy to use", "The button should be green". Gaps also quickly become apparent here. All these queries need to be clarified.
The clarified points must of course also be known to the developer. It is therefore all the more important to start creating test cases at an early stage.
The test object for the system testing is the integrated overall system. I.e. the software or application is tested via the user interface or the interfaces. For a final evaluation, the system must therefore also be complete.
Integration with other systems is not part of system testing, but is often referred to as system integration testing.
Test objectives of the system test
The goal of system testing is to check whether the functional and non-functional requirements for the application are met and sufficiently implemented. In practice, some challenges lurk here that must be considered at an early stage (see also typical problems).
Test environment for system tests
Here also lurks a difficulty. For a valid statement on system testing it is necessary that the test environment corresponds to the productive environment. Or at least very similar to it. Especially when it comes to non-functional test types such as performance testing or reliability. And this duplicate infrastructure can be expensive. This is also where a massive advantage of virtual machines and cloud solutions becomes apparent. If the later productive system is a parameterized virtual machine, it can also be cloned inexpensively for the system testing .
Test data management becomes much more demanding from the system test level onwards. Unit and integration tests usually still handle test data in the local environment of the test case. At system testing , however, significantly more extensive test data sets may be required. E.g. created contracts, historical data, linked data sets, etc. Two procedures have become established in practice:
- Real data: If available, data extracts can be used, for example, from productive systems to use them as test data for the system testing . This is particularly popular during the maintenance phase. Where necessary, this data is anonymized. But always with the awareness of changing the data. The advantage of real data is, of course, its practicality, but also the fact that there are sometimes data and constellations that are not covered by requirements and test cases.
- Synthetic test data: But real data is far from covering all possibilities, corner cases and combinations. Therefore, it is also advisable to generate concrete synthetic data to test special cases: Birthday on 2/29 or postal codes starting with 0 are classics here.
Test automation of the system test
For a long time, test automation at system testing was poor. Tests under the hood, such as unit and integration tests, were well supplied with tools simply because of the proximity to development. With the advent of agile projects, however, there has been a massive push here at the system test level. Both in test automation solutions for surface tests, web tests and also interface tests. There is a wealth of possibilities here today. I have summarized how to implement test automation here in the book "Basiswissen Testautomatisierung".
system testing in agile projects
The model of test levels comes from a time long before agile projects. And is therefore often ignored in Scrum and Co. But if we take a closer look at the model, we see how important the ideas and aspects of the test levels are in agile contexts. Like at system testing: Test the system from the outside. Prepare test data. Consider test cases. And of course set up test automation.
In this context, the model of the test pyramid appears again and again. It provides a similar perspective to the test levels.
Therefore, it is worthwhile to take a look through system testing glasses in agile software development as well. The intention of the system test can be transferred. And use what is relevant in your own project.
When I look at the projects of the last few years, the same challenges in everyday life occur over and over again at system testing :
- Non-functional requirements are not considered. system testing does not only include functional requirements. Non-functional requirements must also be considered here, such as usability tests, performance or reliability tests.
- The system testing is the first test level, which is completely based on the requirements. And from requirements to code and test cases, there is inevitably a media break. Natural language becomes technology. While development can creatively fill in the gaps, a tester cannot in order to create test cases. Here, the same target image must be worked on again and again.
system testing in practice
In practice, the system testing is often the test level to start with. This is because this test level is easier to grasp for departments, testers and management. Take requirements and test the system against them. This is easier to understand than arguing for some small-scale unit tests at code level.
In this situation, however, the system testing often shows the gaps of the other test levels. Instabilities and inconsistent behavior of the software then indicate that there is a lack of robustness in the substructure. Due to the late testability at system testing , these findings unfortunately come very late.
Therefore, it is particularly important to establish the other test levels as well. Only one system testing is too little.
Even if the execution of the system test is possible quite late, the test cases, for example, can already be designed before the first line of code is written. Questions and ambiguities are thus resolved at an early stage.