Automotive testing is the quality assurance of software and hardware in vehicles under strict standards such as Automotive SPICE, functional safety according to ISO 26262, and AUTOSAR. The standards prescribe processes, documents, roles and specific test methods. Safety-critical systems such as an airbag or a windshield wiper require mandatory, clearly defined black-box techniques, and higher risk classes make development and certification considerably more expensive.
Key Takeaways
- Automotive testing is regulated by standards such as Automotive SPICE and functional safety: they define the minimum set of processes, documents, roles and test methods a team has to use.
- When a system’s safety risk rises by one level, the total development effort grows by a factor of two to three, which explains why certified subsystems are hardly ever touched again.
- Even a windshield wiper counts as a safety-critical system in the automotive world, so teams must use prescribed black-box techniques such as equivalence partitioning and boundary value analysis, plus MC/DC coverage.
- Test benches with real hardware run day and night so that as close to one hundred percent of all test cases as possible are automated, because the sheer number of individually configured vehicle variants rules out manual testing.
- The Automotive Tester certification requires the ISTQB Foundation Level, but most of its content is new ground: Automotive SPICE, functional safety and AUTOSAR.
What Sets Automotive Testing Apart From Classic Software Testing
In automotive testing, you always have to think about the hardware too. Even if you work purely as a software tester, you need a basic understanding of the hardware and you test at least part of the hardware communication. That is the first big difference from testing in a web or pure software environment.
A car is big, complex and full of software. Many people, companies and suppliers work on it in parallel. This distributed development is exactly what makes communication hard, and it explains why the industry relies so heavily on norms and standards.
For testers, that means a lot of contact with companies from all over the world and a steep learning curve. The price is the traceability effort. Every activity has to be documented and verifiable.
Automotive Testing Is a Heavily Regulated Environment
The standards govern almost everything a tester does. They define which activities take place, which documents are produced, and which roles have to be involved at which point.
Once functional safety comes into play, the standards go further still. Functional safety covers cases where a software defect could injure a person. For such systems, the standards prescribe the minimum test methods that must be applied.
These requirements are detailed and strict. In practice, teams rarely do more than the standards demand, because the requirements already fill most of the available scope.
Creativity in Testing Is Mandatory Here
In automotive, exploratory and intuitive testing is not an extra. It is required. Testers are obliged to work with the system freely at the end and try things out.
The test process starts differently, though. First come strict methods across many test levels, from black-box techniques to coverage criteria. That baseline workload is fixed.
Only then comes the exploratory part. This mix of prescribed methodology and prescribed creativity is typical of the industry.
How the Development and Test Process Is Structured
Above everything sits a system life cycle with six phases. At the start, neither product nor idea exists; at the end, the product no longer exists. In between come concept, development including testing, production, operation, maintenance and decommissioning.
Testers work almost exclusively in the development phase. It usually follows a V-model, but not the classic four-level software V. In automotive, the V often has seven, nine or even more levels.
These levels break the system down from top to bottom: from the vehicle level through the interior electronics and the ECU level down to the software level. Depending on where you sit, you test small software components, entire ECUs or the integrated system on purpose-built test benches.
On the left side of the V, someone always has to fill the tester role. That covers reviews and static analysis. The role is a mandatory part of the process, even if in practice it is spread across many people.
In Automotive, You Can Pick Your Focus
The range of work is wide. You can spend a whole year testing only on the nearly finished vehicle, adapting and running test cases for every new ECU software version.
If you work closer to the software, you run unit tests, meaning software component tests, along with integration tests and sometimes system tests. Reviews are often part of the job too.
That way you can follow a function from a small software component almost all the way to the finished vehicle. How deep or broad your work goes depends heavily on the company and the area.
Why Agile Hits Its Limits With Hardware
Agile methods have reached automotive too. For large companies, SAFe as a scaled framework is the common approach the industry is trying to roll out.
It works well where software dominates. Server-heavy areas with a lot of backend logic, where the software is later flashed to the vehicle over the air, can be developed largely in an agile way. Many companies are happy with that.
Hardware is where it gets hard. Turning a hardware design into a prototype can easily take eight to ten weeks. That doesn’t fit sprint-based approaches well. One way out is to pull hardware development out as a separate epic that runs in parallel.
The world of documents can be seen through an agile lens as well. Mandatory documents aren’t a tiresome side burden. They are work products that deliver value and can be treated as output in Scrum or SAFe. The shift has started, but it isn’t finished.
How Safety-Critical Systems Determine the Test Methods
As soon as a system is safety-critical, the standards dictate what has to be tested at a minimum. Many of these systems are more critical than you might first expect.
A windshield wiper already counts as a high-risk system. If it fails in the rain, things can get dangerous. An airbag ranks considerably higher still in the risk assessment.
For systems like these, the chain of effects is analyzed and the system is designed to be as modular as possible to limit side effects. Tables then define the mandatory methods. For the windshield wiper, for example, that means black-box techniques such as equivalence partitioning and boundary values at the software unit level, plus MC/DC (modified condition/decision coverage).
Anyone who ignores these requirements and causes damage is liable. Still, the tables leave some room: often you have to pick a suitable combination of several methods.
Model-Based Testing Is Mostly Optional, but Growing
Model-based testing is recommended in many cases, but it isn’t mandatory. For safety-critical applications, the approach is often dropped again.
For topics that aren’t safety-critical, teams have more freedom in choosing methods, as long as the result is considered safe enough. That’s where interest in model-based approaches is growing.
The approach shows up across all areas. In electric vehicles, for example, the energy system is modeled and test cases are generated automatically from the model. Activity diagrams and state machines are the most popular.
At the start of a project, a team of test managers and functional safety managers decides which methods to use. That decision gets documented.
“Anything that isn’t documented never happened in the automotive world.”
(Christian Schwarzer)
New Development Uses Modern Methods, Legacy Remains Untouched
The classic combustion engine world carries a lot of legacy. The software is often older, and following the “never change a running system” principle, certified code that works is hardly ever touched. Old documentation and old code get dragged along.
New development looks different. Electric vehicles and highly automated driving use current methods, and a lot of the work there is model-based.
Existing systems don’t get models retrofitted. The effort wouldn’t pay off, because models created after the fact would have to be verified again as well, without any real added value.
There’s a hard cost logic behind this. When a subsystem moves up one risk level, the whole development process becomes roughly two to three times more expensive. An airbag sits at the highest level, ASIL D. These factors multiply across the levels, which is why working systems are deliberately left alone.
Why Test Automation Is Indispensable in the Automotive Industry
The number of variants forces automation. A configured new car, with its particular combination of headlights, interior lighting and options, often exists only once in the entire world.
In theory, every one of those combinations would have to be tested. In practice that’s impossible, so automation is used on a massive scale. At the unit and code level, it’s easy to set up.
At higher levels, teams build test benches with real hardware that simulate an ECU’s later environment as accurately as possible. Here too, the goal is to automate nearly all test cases. Up to 20 test benches run day and night, backed by purchased server capacity for the software.
Manual testing mainly happens at the very top, on the vehicle itself, for example for driving dynamics. Automation takes the human factor out of repetition and regression testing, and it makes the work more varied at the same time, because the focus shifts to automation and new features.
What the Automotive Tester Certification Covers
The Automotive Tester course usually takes two days and teaches the terms and standards testers need in the industry. It was created with the involvement of the German Testing Board.
It centers on three standards. Automotive SPICE describes the process framework: which documents are produced, who is involved where, plus the test and review processes.
The second block is functional safety, meaning how to deal with systems where a software defect could injure someone, seen from the tester’s perspective. The third standard is AUTOSAR, the communication layer in the vehicle. Think of it as a Lego baseplate: software blocks dock on top, hardware blocks underneath, and both stay replaceable.
The Foundation Level is required for the exam. Most of the content is new, though. If you come from a testing background and aren’t aiming for the certificate, you can take the course without the Foundation Level. For anyone new to the field, the Foundation Level provides a useful grounding in methods and terminology.
Frequently Asked Questions
Do software testers in the automotive sector need hardware knowledge?
Yes. Anyone who performs testing in the automotive environment must take hardware into account, even as a pure software tester. A basic understanding of hardware is part of the job, as is testing hardware communication, at least in part. This is the most significant difference compared to testing in web-based or purely software-based environments. Added to this is the effort required for traceability: every activity must be verifiable.
Is exploratory testing even allowed in highly regulated automotive projects?
It is not only allowed, it is mandatory. Testers are required to work freely with the system and experiment with it toward the end of the process. Before that, however, there is a fixed baseline: strict methods across many test levels, ranging from black-box testing to coverage criteria. The combination of prescribed methodology and mandatory creativity is typical of this industry.
Can automotive projects involving hardware be developed using agile methods?
Only to a limited extent. In server-heavy areas with a lot of backend logic (where the software is later flashed remotely onto the vehicle), agile development works well; large companies rely on SAFe for this. On the hardware side, it can easily take eight to ten weeks to produce a prototype, which doesn’t fit well with sprints. One workaround is to run hardware development as a separate epic in parallel.
Is a windshield wiper considered a safety-critical system?
Yes, it’s classified as a high-risk system because a failure during rain can be dangerous. For this system, the standards require black-box methods at the software unit level, such as equivalence partitions, boundary value analysis, and MC/DC. An airbag is rated significantly higher in risk assessment. Anyone who disregards the requirements and causes damage is liable.
Is model-based testing mandatory in the automotive sector?
No, it is recommended in many cases but not mandatory. For safety-critical applications, this approach is often dropped; for non-safety-critical areas, there is greater freedom in choosing methods. Activity diagrams and state machines are primarily used, for example, to model the energy system in electric vehicles and automatically generate test cases from it.
Why is existing, certified vehicle software rarely modernized?
Because the cost considerations make it unfeasible. If a subsystem moves up one risk level, the entire development process becomes roughly two to three times more expensive, and these factors multiply across the levels. An airbag is at the highest level, ASIL D. Consequently, existing certified code remains untouched, and outdated documentation is carried over.
Why is automation essential for testing in the automotive sector?
The sheer variety of configurations leaves no choice. A configured new car, with its unique combination of headlights, interior lights, and options, often exists as a one-of-a-kind model worldwide; theoretically, every possible combination would have to be tested. At the unit and code levels, automation is easy to implement; at higher levels, up to 20 test benches with real hardware run day and night. Manual testing is primarily performed on the vehicle’s top-level systems, such as driving dynamics.
Is the ISTQB Foundation Level required for certification as an automotive tester?
The Foundation Level is mandatory for the exam. You can also attend the two-day course without it if you come from a testing background and are not seeking certification. In terms of content, the topics are largely new: Automotive SPICE as a process framework, functional safety from a tester’s perspective, and AUTOSAR as the in-vehicle communication layer. For beginners, the Foundation Level provides the foundational knowledge of methods and terminology.
