Automotive testing refers to the quality assurance of software and hardware in vehicles under strict standards such as automotive SPICE, functional safety and AUTOSAR. Processes, documents, roles and specific test methods are prescribed. Safety-critical systems such as airbags or windshield wipers require mandatory defined black box procedures; higher risk classes make development and certification significantly more expensive.
Key Takeaways
- Automotive testing is regulated by standards such as automotive SPICE and functional safety: they stipulate the minimum processes, documents, roles and test methods that must be used.
- If the security risk of a system increases by one level, the total development effort increases by a factor of two to three, which explains why certified subsystems are hardly ever touched.
- Even a windshield wiper is considered a safety-critical system in the automotive environment, forcing teams to use prescribed black box methods such as equivalence partitions, boundary value analysis and MCDC.
- Test benches with real hardware run day and night to ensure that as close to one hundred percent of all test cases as possible are automated, because the sheer number of variants of individually configured vehicles precludes manual testing.
- The ISTQB Foundation Level is a mandatory prerequisite for certification as an automotive tester, but the content mainly covers new topics such as automotive SPICE, functional safety and Autosar.
What distinguishes automotive testing from classic software testing
Testing in the automotive sector always means thinking about the hardware as well. Even if you work purely as a software tester, you need to know the basics about the hardware and test at least some of the hardware communication. This is the first major difference to testing in a web or pure software environment.
A car is large, complex and full of software. Many people, companies and suppliers work on it in parallel. It is precisely this distributed development that makes communication difficult and explains why the industry works so strongly with norms and standards.
For testers, this means a lot of contact with companies from all over the world and a high learning factor. The price for this is the tracking effort. Every activity must be verifiable.
Automotive testing is a highly regulated environment
The standards regulate almost everything for testers. They define which activities are carried out, which documents are created and which roles must be involved and where.
As soon as functional safety comes into play, the standards go even further. Functional safety describes the case in which a software error could injure a person. For such systems, the standards stipulate the minimum test methods to be used.
These specifications are detailed and strict. In practice, teams rarely do more than what the standards require because the specifications already largely fill the scope.
Creativity in testing is required here
Exploratory and intuitive testing is not an add-on in the automotive environment, it is mandatory. Testers are obliged to work freely with the system and try it out.
The start of the test process is different. At the beginning, there are strict methods across many test levels, from black box to coverage procedures. This basic load is predetermined.
Only then comes the exploratory part. The combination of fixed methodology and prescribed creativity is typical for this industry.
How the development and test process is structured
Above all, there is a system life cycle with six phases. At the beginning, neither the product nor the idea exists; at the end, the product no longer exists. In between are conception, development including testing, production, use, maintenance and decommissioning.
Testers work almost exclusively in the development phase. A V-model is usually used there, but not the classic software V with four stages. In the automotive industry, the V model often has seven, nine or more stages.
These levels break down the system from top to bottom: from the vehicle level to the interior electronics to the control unit level to the software level. Depending on the position, small software components, entire control units or the entire system are tested on specially constructed test benches.
On the left-hand side of the V, there must always be someone in the tester role. This involves reviews and static analysis. This role is a mandatory part of the process, even if in practice it is spread across many people.
In the automotive sector, you can choose your focus
The range of activities is broad. You can spend a year testing exclusively on the almost finished vehicle, adapting and carrying out test cases for each new ECU version.
If you work closer to the software, you will carry out unit tests, i.e. software component tests, as well as integration and sometimes system testing. Reviews are often added.
In this way, a function can be accompanied from the small software component to almost the finished vehicle. How deeply or broadly you work depends heavily on the company and the area.
Why agility has its limits when it comes to hardware
Agile methods have also arrived in the automotive sector. For large companies, SAFe as a scaled framework is the common approach that the industry is trying to implement.
It works well where software dominates. Server-heavy areas with a lot of backend logic, where the software is later flashed remotely onto the vehicle, can be developed largely agilely. Many companies are happy with this.
Hardware is where it gets difficult. It can quickly take eight to ten weeks for a prototype to emerge from hardware development. This does not fit in well with sprint-based approaches. One way out is to extract hardware development as a separate epic in parallel.
The document world can also be thought of as agile. Mandatory documents are not an annoying accompanying burden, but work products that deliver value and can be categorized as output in Scrum or SAFe. The change has been initiated, but not completed.
How safety-critical systems determine test methods
Once a system is safety critical, the standards dictate what must be tested as a minimum. Many of these systems are more critical than one might initially expect.
A windshield wiper is already considered a high-risk system. If it fails in the rain, it can be dangerous. An airbag has an even higher risk assessment.
For such systems, the chain of effects is analyzed and the system is designed as modularly as possible in order to limit side effects. Tables then define the mandatory methods. For the windscreen wiper, for example, black box methods such as equivalence partitions and boundary values at software unit level as well as MC/DC, Modified Condition/Decision Coverage.
Anyone who disregards these specifications and causes damage is liable. However, there is still room for maneuver in the tables: a suitable combination of several methods must often be selected.
Model-based testing is mostly optional, but on the rise
Model-based testing is recommended in many cases, but is not mandatory. The approach is often removed again for safety-critical applications.
For non-safety-critical topics, the choice of method is freer as long as the result is considered safe enough. Interest in model-based approaches is growing here.
The approach is used across all areas. In the case of electric vehicles, for example, the energy system is modeled and test cases are automatically generated from this. Activity diagrams and state machines are the most popular.
At the beginning of a project, a team of test managers and functional safety managers decide on the choice of method. This decision is documented.
Anything that is not documented has not taken place in the automotive environment.
Christian Schwarzer
New development uses modern methods, passed remains untouched
There is a lot of legacy in the classic combustion world. The software is often older, and according to the principle “never change a running system”, certified, running code is hardly ever touched. Old documentation and old code are dragged along.
New developments look different. The latest methods are used for electric vehicles or highly autonomous driving, where a great deal of model-based work is carried out.
Existing systems are not retroactively given models. The effort would not be worth it, as models created retrospectively would also have to be checked again, without any real added value.
There is a hard cost logic behind this. If a subsystem moves up a risk level, the entire development process becomes roughly two to three times more expensive. An airbag is at the highest level, ASIL D. Such factors multiply over the levels, which is why functioning systems are deliberately no longer touched.
Why test automation is indispensable in the automotive industry
The diversity of variants forces automation. A configured new car with its combination of headlights, interior lights and options often only exists once in the world.
In tester terms, each of these combinations would have to be tested. In practice, this is impossible, which is why there is massive automation. This is easy to implement at unit and code level.
At higher levels, test benches with real hardware are created that simulate the later environment of a control unit as accurately as possible. Here too, the aim is to automate almost all test cases. Up to 20 test benches run day and night, supplemented by purchased server capacity for the software.
Manual testing is mainly carried out at the very top of the vehicle, for example for driving dynamics. Automation removes the human factor from repetitions and regression testing and at the same time makes the work more varied because the focus is on automation and new features.
What the automotive tester certification teaches
The Automotive Tester course usually lasts two days and teaches the terms and standards that testers need in the industry. It was set up with the participation of the German Testing Board.
The focus is on three standards. Automotive SPICE describes the process landscape: which documents are created, who is involved and where, as well as the testing and review processes.
The second block is functional safety, i.e. dealing with systems in which a software error could injure someone, viewed from the tester’s perspective. The third standard is AUTOSAR, the communication layer in the vehicle. You can think of it like a Lego board, with software blocks at the top and hardware blocks at the bottom that remain replaceable.
Foundation level is mandatory for the exam. In terms of content, however, the topics are mostly new. If you come from a testing environment and are not aiming for certification, you can also attend the course without Foundation Level. For all newcomers, the Foundation Level provides a useful foundation of methods and concepts.
