Quality assurance has to be effective, and it has to be efficient. Automation is the obvious way to gain efficiency, and test automation has long outgrown its early days: it is now a must-have, and it reaches far beyond running tests. At the same time, new challenges keep coming. Higher efficiency, better integration into agile and continuous approaches, and more complex systems with more interfaces, combined with demands for safety, security and robustness, are just some of the aspects test automation engineers have to deal with today.
Alongside designing and writing software, quality assurance is one of the fundamentals of software development. Testing is the most widely used form of quality assurance: it compares how the software actually behaves with how it is expected to behave.
That comparison can take a lot of effort. Regression testing after small changes to the software is especially costly by comparison. Regression tests have to be run, for example, to prove that changes to the software don’t cause unwanted side effects. Because they deliberately also cover functionality that the change is not supposed to touch, it quickly becomes obvious that the share of tests you have to run, out of all tests, is larger than the share of changed code out of the whole system. Testing can therefore take a lot of effort compared with building the software itself, especially when it is done manually.
Since running tests, and regression tests in particular, is largely repetitive work, automating their execution is the natural way to become more efficient.
From Nice-to-Have to Must-Have
For a long time, automated test execution was an add-on to manual testing, sometimes little more than a gimmick. Teams picked suitable test cases and used automation tools to record or implement test scripts for them. That worked more or less well, and in the long run it was hard to maintain. Often it took the initiative of individuals to keep tools and scripts usable at all.
Those days are behind us. Paradigm shifts such as agile development and the growing complexity of the systems being built have made test automation indispensable. Many projects now run automated tests at every level with a wide range of tools and increase test coverage step by step. This has also brought a clear leap in tool support: test automation tools are more stable and easier to use, test scripts are easier to maintain, and the tools adapt better to what test automation engineers need.
Test Automation Is More Than Test Execution
People often equate test automation with automated test execution. But there is also automated test design. As the number of test cases grows, so does the cost of maintaining the test scripts, and to keep that manageable, test design has to be automated too. Additional layers of abstraction hide what doesn’t matter, and test scripts can be generated automatically from models.
Even that doesn’t come close to exhausting the potential that automation offers testing today. Across the whole process there are countless activities that can be created, run or checked with full or partial automation. In the process, the lines between testing, quality assurance and development are blurring more and more. You’ll find a few examples in the section “Examples of Activities You Can Automate” below.
Test Automation Brings People Together
In our experience, agile projects show an interesting picture. Test automation has to be part of the work from day one, because otherwise high software quality can’t be sustained. But there is a social side to it as well. The formalization required up front gets different participants, including ones who (so far) worked on their own, talking about a shared topic earlier than before. This matters most in hybrid project structures, for instance when the business department isn’t part of the team. You could go so far as to call test automation a team-building exercise.
Conclusion
By 2018, when this article was written, test automation had clearly left its infancy behind, even if there was still plenty of room to grow. Test automation was finally delivering on its promises of efficiency and repeatability, and it had become much easier to use and to maintain. Not least, projects can use test automation to bring roles into contact that previously worked largely on their own, for example to decide where automation makes sense from a risk perspective or adds value.
Even though support for automation has improved considerably in recent years, pitfalls remain. Incompatibilities, interface problems, unstable environments or poor data quality can drive up the effort for creating and maintaining test scripts. So always keep one eye on the benefits and on the risk assessment, so that test automation doesn’t become an end in itself.
Back then, it was already becoming clear that machine learning and AI would open up entirely new fields and uses for test automation. These mechanisms will be needed to keep software and data quality high. One thing has already happened: often nobody has a full view of the software, the architecture or the data it needs anymore. That calls for intelligent systems that present this information in an understandable way and detect inconsistencies automatically.
Examples of Activities You Can Automate
Static Analysis
- Code analysis
- Artifact checks
- Refactoring support
Test Data Management
- Generating test data
- Anonymizing production-like test data
- Populating and clearing databases
- Archiving test data
Test Environment
- Setting up operating systems
- Configuring virtual machines
- Restoring a defined state of the test environment
Deployment
- Quality gates
- Build pipeline
- Deployments and automated tests
Documents
- Generating reports
- Evaluating metrics
- Analyzing requirements
Non-Functional Testing
- Security testing
- Load and performance testing
- Stress testing
- Usability testing
Frequently Asked Questions
Why is the effort required for regression testing so high?
Regression testing deliberately checks functionality that is not even supposed to be affected by a change. As a result, the proportion of tests that need to be run out of the total number of tests exceeds the proportion of the modified software out of the total software. When performed manually, testing can thus become more time-consuming than the software development itself. Since the process consists largely of repetitive tasks, automation is the obvious solution here.
How far had tool support for test automation advanced?
By 2018, tool support had taken a significant leap forward: the tools were more stable and user-friendly, test scripts had improved maintainability, and the tools could be better adapted to the needs of test automation specialists. Previously, automation solutions were often developed on the initiative of individuals, worked more or less well, and had low maintainability in the long run.
Is it enough to automate test execution?
No. As the number of test cases increases, so do the maintenance costs for the test scripts; therefore, test design must also be automated. Additional levels of abstraction filter out irrelevant details, and test scripts can be automatically generated based on models. Anyone who equates test automation with automated test execution is failing to realize its full potential.
What activities, besides test execution, can be automated?
There are candidates throughout the entire process: static analysis with code analysis, artifact checks, and refactoring support; test data management with generation, anonymization of production-like data, as well as populating and emptying databases; setting up and resetting test environments; quality gates and build pipelines; generating reports and evaluating metrics; and non-functional testing such as security, load, stress, and usability tests.
What role does test automation play in agile projects?
It must be integrated from the very beginning; otherwise, high software quality cannot be maintained. There is also a social effect: The formalization required up front brings stakeholders (who previously worked independently) into dialogue with one another at an earlier stage. This is particularly effective in hybrid project structures, such as when the business department is not part of the team.
What drives up the costs of test automation?
Incompatibilities, interface issues, unstable environments, and poor data quality can significantly increase the effort required to create and perform maintenance on test scripts. These pitfalls persist even after improvements in tool support. Therefore, it is essential to continuously assess benefits and risks; otherwise, automation becomes an end in itself.
What are machine learning and AI used for in quality assurance?
In 2018, the article identified a need stemming from the fact that often no one has a complete overview of the software, the architecture, or the required data. There is a need for systems that present this information in an understandable way and automatically detect inconsistencies. Such mechanisms were deemed necessary to maintain high levels of software and data quality, but at that time were only in their infancy.
