Quality assurance must not only be effective, but also efficient. Automation is the tool of choice for increasing efficiency and has long since outgrown its infancy. However, new challenges for quality assurance and test automation in particular are not long in coming: greater efficiency, better integration into agile and continuous approaches, more complex systems with more interfaces coupled with requirements for safety, security and robustness are just a few of the aspects that test automation specialists have to face today.
In addition to the design and writing of the software, quality assurance is also part of the fundamentals of software development. Testing is the most frequently used means of quality assurance and compares the actual behavior of the software with the expected behavior.
However, this testing can be very time-consuming. In particular, the effort required for regression tests after minor changes to the software is comparatively high: regression tests must be carried out, for example, to prove that changes to the software do not have any undesirable side effects. As this involves testing functionality that should not be affected by the changes, it is easy to see that the proportion of tests to be executed in the total number of tests exceeds the proportion of software changes in the total amount of software. The test can be correspondingly time-consuming compared to the software creation, especially if it is carried out manually.
Since the execution of tests and especially regression tests often involves repetitions, the automatic execution of these tests is an obvious way of increasing efficiency.
From freestyle to duty
For a long time, automated test execution was an add-on to manual test execution - sometimes even just a gimmick: suitable test cases were selected and corresponding test scripts were recorded or implemented with the help of automation tools. This worked more or less well and was less than maintainable in the long run. Solutions were often created on the initiative of individuals so that tools and test scripts could still be used.
Those days are behind us. Not least due to paradigm shifts such as agile software development or the increasing complexity of the systems developed, test automation is indispensable today. In many projects, tests are automated across all levels using a wide variety of tools and test coverage is gradually being increased. This has also led to a significant leap in tool support: Test automation tools today are more stable, offer greater convenience, provide easier maintainability of test scripts and can generally be better adapted to the needs of test automation engineers.
Test automation is more than test execution
The term test automation is often equated with automated test execution. Of course, there is also automated test design. In order to keep the maintenance costs for the test scripts manageable as the number of test cases increases, the test design must also be automated: Additional abstraction levels are used to abstract from non-essentials - test scripts can be generated automatically on the basis of models.
However, the potential that automation offers for testing today is far from exhausted - there are countless activities throughout the entire process that can be created, executed or tested automatically or semi-automatically. The boundaries between testing, quality assurance and development are becoming increasingly blurred. A few examples can be found in the box.
Test automation as an integration factor
In our experience, agile projects in particular present an interesting picture. Here, test automation must be practiced right from the start, as otherwise high software quality cannot be maintained. However, test automation also has a social aspect here, as the necessary formalization in advance encourages various participants, including those who have (previously) worked independently, to discuss a common topic earlier than before - especially when hybrid project structures prevail, where, for example, the specialist department is not part of the team. In this way, test automation can even be seen as a team-building measure.
Fazit
As of 2018, test automation has clearly outgrown its infancy, even if there is still plenty of room for improvement. Test automation now manages to fulfill the promises of efficiency and repeatability and has become much simpler and easier to use. Last but not least, test automation can be used in projects to bring different roles into contact and discussion that have previously acted rather autonomously. For example, to determine where automation makes sense from a risk perspective or contributes to value.
However, even though support for automation has improved significantly in recent years, there are still pitfalls lurking: incompatibilities, interface problems, unstable environments or a lack of data quality can increase the time and effort required to create and maintain test scripts. The benefits and risk assessment must therefore always be kept in mind so that test automation does not become an end in itself.
The next few years will be very exciting here. The use of machine learning and artificial intelligence opens up completely new fields and potential applications that are only just beginning to be exploited. The fact is that these mechanisms will be needed to keep the quality of software and data high. Because one thing has already happened: Often, no one has an overview of the complete software, the architecture or the required data. This calls for intelligent systems that present this information to us in an understandable way and automatically uncover contradictions.
Examples of activities that can be automated
Static analysis:
- Code analysis
- Artifact checks
- Support with refactoring
Test date management:
- Generating test data
- Anonymization of production-related test data
- Filling and emptying databases
- Archiving test data
Test environment:
- Setup of operating systems
- Configuration of virtual machines
- (Re)establishing a defined state of the test environment
Provision:
- Quality-Gates
- Build-Pipeline
- Deployments und automatisierte Tests
Documents:
- Generating reports
- Evaluating metrics
- Analyzing requirements
Non-functional tests:
- Security tests
- Load and performance tests
- Stress tests
- Usability tests
Frequently Asked Questions
Why is the effort required for regression testing so high?
Regression testing deliberately checks functionality that is not even supposed to be affected by a change. As a result, the proportion of tests that need to be run out of the total number of tests exceeds the proportion of the modified software out of the total software. When performed manually, testing can thus become more time-consuming than the software development itself. Since the process consists largely of repetitive tasks, automation is the obvious solution here.
How far had tool support for test automation advanced?
By 2018, tool support had taken a significant leap forward: the tools were more stable and user-friendly, test scripts had improved maintainability, and the tools could be better adapted to the needs of test automation specialists. Previously, automation solutions were often developed on the initiative of individuals, worked more or less well, and had low maintainability in the long run.
Is it enough to automate test execution?
No. As the number of test cases increases, so do the maintenance costs for the test scripts; therefore, test design must also be automated. Additional levels of abstraction filter out irrelevant details, and test scripts can be automatically generated based on models. Anyone who equates test automation with automated test execution is failing to realize its full potential.
What activities, besides test execution, can be automated?
There are candidates throughout the entire process: static analysis with code analysis, artifact checks, and refactoring support; test data management with generation, anonymization of production-like data, as well as populating and emptying databases; setting up and resetting test environments; quality gates and build pipelines; generating reports and evaluating metrics; and non-functional testing such as security, load, stress, and usability tests.
What role does test automation play in agile projects?
It must be integrated from the very beginning; otherwise, high software quality cannot be maintained. There is also a social effect: The formalization required up front brings stakeholders—who previously worked independently—into dialogue with one another at an earlier stage. This is particularly effective in hybrid project structures, such as when the business department is not part of the team.
What drives up the costs of test automation?
Incompatibilities, interface issues, unstable environments, and poor data quality can significantly increase the effort required to create and perform maintenance on test scripts. These pitfalls persist even after improvements in tool support. Therefore, it is essential to continuously assess benefits and risks; otherwise, automation becomes an end in itself.
What are machine learning and AI used for in quality assurance?
In 2018, the article identified a need stemming from the fact that often no one has a complete overview of the software, the architecture, or the required data. There is a need for systems that present this information in an understandable way and automatically detect inconsistencies. Such mechanisms were deemed necessary to maintain high levels of software and data quality, but at that time were only in their infancy.
