How Types of Systems Shape Test Automation
Different types of systems make different demands on test automation. The system type determines the automation approach, among other things, and it narrows down which tools you can use at all.
A desktop application with a single user can be automated largely through its user interface. A client-server system raises the question of whether the GUI needs automated tests at all and how to simulate several users at once. Complex systems such as data warehouses have no user interface and can hardly be tested manually, so they need a combination of automated approaches. Mobile apps add device variety, interruptions and changing networks. Embedded systems bring safety standards that classify the test tools themselves, and cloud-based systems spread responsibility across several parties. Web applications call for an early decision on which browsers and browser versions to cover. The sections below go through the most common system types one by one.
Desktop Applications
For a long time, pure desktop applications were the only thing there was to automate. They usually consist of software alone and don’t communicate with other systems. That means they hardly have to deal with constraints or interfaces, and developers can focus entirely on what the software is meant to do.
Alongside developer tests at the component level that focus on the application’s own functionality, automated testing through the user interface makes sense. You can use any tool that supports the technology of the application or its user interface. You may also need to read or create test data through the file system, and possibly in the formats the application uses. Further integration levels are not needed, at least at the application level. The internal components are, of course, still integrated with each other according to an integration strategy. Sometimes it helps to build helper functions into the application for testing purposes, for example to prepare tests or to check results.
Automation is considerably simpler for this kind of system because its dependencies on other software systems are limited. A small set of interfaces is very likely enough to automate a functional test.
Because this system type is inherently designed for one user at a time, a whole series of automation problems never comes up. You can skip tests with several concurrent users, for example. You should not skip tests with several instances running in parallel, though.
Client-Server Systems
The rise of client-server systems was the first step up in complexity for test automation. Data is kept centrally on a server, which can also be responsible for large parts of the system’s functionality. To tell the variants apart, people distinguish between a fat client and a thin client. As the names suggest, a fat client holds much more of the functionality, while a thin client is closer to a screen for display and input, with most of the functionality staying on the server.
When to Skip Automated GUI Tests
A key decision when automating tests for these systems is whether to include the user interface or whether a manual test is more efficient for it. With thin clients in particular, it can be enough to let the automation work directly against the client interfaces and skip automated GUI tests altogether. One advantage is that the automated test cases usually run much faster. With fat clients, this approach is usually not appropriate, because a large part of the functionality lives in the client and would go untested.
Simulating Multiple Concurrent Users
In most client-server systems, several users work with the system at the same time. For an automated test, there are several possible setups:
- Several users on the same test machine
- Several users on different physical machines
- Several users on different virtual machines
In the first case, the automated test cases have to run in parallel on one physical machine. Most test tools offer no explicit support for this, because automation through the GUI needs a certain degree of exclusive access. Building that capability yourself, specifically for this purpose, can take a great deal of effort.
The second option makes the test environment hard to control. Configuring and maintaining several physical machines for automation takes considerable effort, even when they are all the same model.
The most widely used option is testing on several virtual machines. Its advantage is that you can simply replicate one defined machine configuration across many instances.
Experience shows that many test teams using a virtualized environment for the first time badly underestimate the administration work and effort involved. Effective configuration management for the virtual test machines is critical to success.
Web Applications
A special case of client-server applications has become very widespread: web applications. There is usually no application-specific client, just a generic one, the browser. Because the transmitted data is highly standardized (HTTP and HTML), you can use methods that target these protocols and take advantage of them, such as capture and replay at the protocol level. Many tools explicitly support web applications.
Running tests in parallel is also easier with web applications. Some tools access the application’s interface through JavaScript rather than the physical GUI, or run their tests entirely at the level of the underlying protocols and formats, which usually shortens test execution considerably.
One decision that makes choosing a tool much easier is whether automated tests need to run on different browsers and browser versions. The answer depends mainly on the functionality provided inside the browser, meaning JavaScript, Ajax and similar techniques.
Semi-automated tests are another way to run automated test sequences across several browsers. Automated functional tests run and capture screenshots along the way, and after the run a tester reviews the screenshots by hand to check that everything displays correctly. This can strike a good balance between manual testing effort and automation effort, because machine verification of how web pages with dynamic content are displayed is not yet reliable, at least at the time of writing.
Test Automation for Mobile Applications
Conceptually, automating tests for mobile applications is similar to testing client-server systems. The difference is that mobile devices, rather than desktop clients, communicate with the server. So why look at mobile apps separately at all? Despite the conceptual similarity, this area brings some specific challenges that justify a closer look. This section covers the main ones.
Choosing the Test Platforms
The market offers a large number of devices that could matter for a test automation project. Selecting a sensible subset of devices for test execution is therefore a major problem.
Handling Interruptions
In mobile applications, interruptions such as incoming calls, text messages and push notifications are a real challenge for tool vendors and test automation engineers alike. If you run the tests on emulators or simulators, these interruptions are relatively easy to simulate. On physical devices, that is hard to do.
Different Device Hardware
Mobile devices in particular come with a wide range of hardware. The variety of screen sizes, resolutions and pixel densities exists in this form only on mobile devices. These factors matter much less when automating desktop applications.
Network Performance and Connection Types
Mobile applications often have to work with different and constantly changing network connections, which raises the question of how to cover this in test automation. In practice, these tests are either done manually as field tests or in a test environment with simulated network connections (WAN emulators).
Test Automation for Embedded Systems
Embedded systems are defined by software that is embedded in hardware. The system under test is therefore not just the software but hardware and software together. When several components with different hardware dependencies interact, developing and testing embedded systems can get as complex as you like. Additional factors, such as interacting third-party systems or data dependencies, add even more.
Embedded systems are also often safety-critical, meaning that important assets such as property or human life depend on their behaving correctly. There are standards for this, such as IEC 61508, and domain-specific ones such as ISO 26262 for the automotive industry or EN 50128 for the railway sector.
Tool Classes in EN 50128
These standards contain method tables for the type of test approach as well as classifications for the tools used. EN 50128, for example, distinguishes three classes:
- T1: tools with no influence on the test object
- T2: verification and test tools whose errors could cause defects to go undetected
- T3: tools that have a direct influence on the test object
Test automation tools fall under T2, which means they need their own separate quality evidence.
Data Warehouses
Data warehouses are an example of complex systems with many interfaces, a lot of data and often a structure that is anything but intuitive. Databases and the underlying systems and standard products don’t differ much from other applications when it comes to testing: they have concrete requirements and use cases. A data warehouse is different. It is a central collection of data from several of a company’s systems, structured and prepared so that it allows comprehensive analysis in support of management and business decisions.
A data warehouse (DWH) essentially follows two principles:
- Integration of data
- Separation of data
Running a DWH, from acquiring the data and storing it in the DWH database through to managing data sets for later analysis and reporting, is called data warehousing.
Why Manual Testing Is Nearly Impossible
Apart from the organizational issues and the infrastructure (large volumes of data, legally sensitive data and so on), several aspects make manual testing of a data warehouse almost impossible:
- Many technical interfaces to many source systems
- No graphical user interface
- Complex core functionality such as data historization or consistency checks
- Import, export and data semantics are complex, and the test team often doesn’t know them in detail
How to Automate Data Warehouse Tests
A thorough test usually needs a combination of automated approaches. For the core of the data warehouse, meaning the central data storage and the actual warehouse functionality such as historization and references, there are usually consistency rules the data in the core system has to satisfy. An automated system can check whether these rules actually hold, for example whether the most recent record in the data history always carries the flag for the currently valid record.
Other approaches to testing a DWH include:
- Plausibility checks: import valid and invalid records. The target system must accept the valid records, and the invalid ones must show up as “rejected” in the log.
- Reimplementation: part of the system’s functionality is reimplemented in the automation framework for testing purposes.
- Defined input-output pairs: a known set of test records, derived from the import and transformation rules with test design techniques, is imported, and the result is compared with known expected data derived the same way.
Cloud-Based Systems
The defining feature of cloud computing is that users no longer provide or run applications, tools, development environments, management tools, storage, networks, servers and so on themselves. Instead, they rent them from one or more providers who offer the IT infrastructure publicly as cloud services over a network. For users, that removes the cost of buying and maintaining IT infrastructure. They pay only for the services they actually use, for as long as they use them. Standardized, highly scalable services let many companies use services that used to be almost unaffordable.
One issue cloud users have to deal with is data security. It is their responsibility to decide which data they hand over outside the company, and how much of it.
Outsourcing operating environments and relying on external services for functionality also has a major impact on testing. In these multi-layered setups, responsibility no longer lies with a single party, and parts that matter for functionality are developed and operated independently of each other. That makes continuous checking of the systems’ functionality necessary, in the form of frequent regression testing.
What matters here is a clearly defined scope for testing and test automation. Tests a cloud infrastructure provider has to run generally have a very different focus from tests run by a platform or software developer, or by the customer.
Frequently Asked Questions
Is it necessary to automate tests with multiple concurrent users for desktop applications?
No. Such systems are inherently designed for a single concurrent user, so there is no need for tests with multiple users running in parallel. However, you should not skip testing multiple instances running in parallel. It makes sense to perform developer tests at the component level, combined with automation via the user interface, occasionally supplemented by built-in helper functions for test preparation and result verification.
Is automated testing via the user interface worthwhile for client-server systems?
That depends on how much functionality is contained in the client. With thin clients, it is often sufficient to run automation directly through the client interfaces and forego GUI testing, which significantly increases execution speed. With fat clients, this is not advisable because a large portion of the functionality resides in the client and would remain untested.
How can a test with multiple concurrent users be set up technically?
There are three ways: multiple users on the same test machine, on different physical machines, or on virtualized machines. The first approach often fails due to limitations in the tools, since GUI automation requires a certain degree of exclusivity. Physical machines entail configuration and maintenance overhead. In 2021, the virtualized approach was the most commonly used, although teams often underestimated the administrative overhead.
Can the correct display of web pages be checked automatically?
In 2021, it was not possible to reliably ensure this automatically for dynamic content. Semi-automated tests offer a middle ground: An automated functional testing tool runs, capturing screenshots along the way, and a tester then manually reviews these to ensure correct display. This strikes a balance between manual testing effort and automation effort.
What makes test automation for mobile applications more difficult than for desktop software?
Conceptually, it is similar to testing client-server systems, but in practice, four additional challenges arise: selecting a meaningful subset of relevant end devices, handling interruptions such as calls, text messages, or push notifications, the variety of hardware, screen sizes, and resolutions, and fluctuating network connections. Interrupts can be easily simulated on emulators, but hardly at all on physical devices.
Do testing tools for safety-critical embedded systems require their own quality evidence?
Yes. The EN 50128 standard for the rail sector classified tools into T1, T2, and T3 in 2021: T1 has no impact on the test object; T2 includes verification and test tools whose errors can cause defects to remain undetected; T3 directly affects the test object. Test automation tools fell under T2, for which separate quality evidence was required.
Why is manual testing of data warehouses virtually impossible?
Several factors make this difficult: numerous technical interfaces to source systems, the absence of a graphical user interface, complex core functionalities such as historization or consistency checks, and data semantics that the test team often does not understand in detail. Therefore, a combination of automated approaches is common: verification of consistency rules, plausibility checks using valid and invalid data records, reimplementation of functionality, and defined input-output pairs.
How does the testing process change when parts of a system are procured as cloud services?
Responsibilities no longer lie with a single party, and functionality-relevant components are developed and operated independently of one another. Therefore, continuous checking in the form of frequent regression testing is necessary. The scope must be clearly defined: Tests conducted by a cloud infrastructure provider generally have a different focus than those conducted by a platform or software developer or by the customer itself.
