Embedded software testing refers to the testing of software that runs closely together with dedicated hardware and fulfills a specific task in a higher-level system. Tests are carried out at several levels: Unit testing, integration testing and system testing, often on real target hardware. Safety-critical systems such as ABS are subject to normative standards, such as MISRA for code quality.
Key Takeaways
- Embedded software cannot be tested separately from the hardware: system testing requires real target hardware because timing and bus protocols cannot be fully mapped in the simulator.
- Functionally safe embedded systems are subject to normative requirements such as the V-model, which makes specification and test verification mandatory at every level.
- Static code analysis against MISRA rules is an established standard in the safety-critical embedded environment, which preemptively excludes an entire class of hardware-related errors in C and C++.
- Continuous integration with automated testing on real target hardware is already standard in professional embedded projects, because otherwise parallel teams have no reliable basis for integration.
- Maintainability and adaptability are often considered too late in embedded projects, which quickly leads to technical debt and time pressure with several device variants.
What makes an embedded system
An embedded system is software that is embedded in a larger system and fulfills a very specific task there. It usually performs this one task very well.
The spectrum is broad. An ABS system in a car is its own embedded system, embedded in the complex overall car system. The coffee machine that makes good coffee is also an embedded system: with just one user who starts it, but with a clearly defined function.
The specific task has a pleasant side effect for development. The requirements change less frequently than in the web world because the function is fixed from the outset. Only when the device reaches the user does new functionality flow in via customer experience feedback, usually at the top of the application level.
Why embedded testing cannot do without hardware
Testing in the embedded environment works closely with the hardware, which is why you ultimately have to test with hardware and in a system context. This is exactly where the biggest difference to pure application software lies.
The device must be tested in its system context, and this brings with it tough boundary conditions. Timing is critical for the ABS system. The signals have to be transmitted to the bus with precise timing, and you want to simulate complex scenarios of the overall system at the same time.
This hardware dependency is the constant that shapes almost every decision in the development and test process. It makes embedded tests more complex than tests that can run completely on a simulated environment.
Quality begins before the first test, in the architecture
Embedded quality is not decided during testing, but in the architecture. Alexander Eisenhuth first looks through the architect’s glasses, and this is where the course is set for maintainability, performance and adaptability.
The resources a device needs depend on which qualities you prioritize. If you focus on maintainability and don’t need top performance, you can choose a generously dimensioned device. If costs are the driving factor, the hardware becomes small, and then suddenly kilobytes of RAM are the limit.
You don’t deal with scarce hardware by desperately saving code, but by making decisions higher up: about the programming language, the libraries, the concept. One weakness that takes its revenge is the lack of adaptability. Anyone who plans device variants but has no architecture concept for them quickly slips into technical debt. Time-to-market is then pressing, and the architecture falls short of what would be possible.
How the code is checked before it runs
Static code analysis catches technical errors in the embedded area that easily happen in hardware-related languages such as C and C++. There are separate standards for functionally safe development.
The MISRA coding standard specifies rules, and static analysis automatically checks whether the source code complies with them. This covers a large proportion of the typical sources of technical errors before anything even runs on the hardware.
In addition, the quality measures that also apply outside of embedded also apply: Complexity of the source code, modules that are too large, spaghetti code. These analyses are not specific to embedded, but are an integral part of it.
Which programming languages are in use today
The basis is classic C, plus C++. Assembler is hardly ever seen in practice.
Rust is interesting for functionally safe systems because the language already has built-in safety features. This fits well with the requirements of safety-critical development.
However, the Rust ecosystem is not yet mature for embedded. The tooling around it, such as code analysis, must first grow before Rust becomes a real standard. The direction is right: There are many indications that Rust will become the next language in this field.
The test levels in the embedded environment
In terms of structure, embedded testing differs little from the classic approach. It involves unit testing, integration testing and system testing, except that the hardware comes into play at several points.
This is what the chain looks like when a product is created from scratch:
| stage | what it’s about |
|---|---|
| Evaluation on E-Val board | Test critical things early on existing CPU and peripherals, implement driver layer |
| Hardware and product tests | EMC tests and temperature tests on own electronics development, software does not have to be ready yet |
| Unit tests | Testing units in advance so that the integration fits together cleanly |
| Integration testing | Component integration without hardware or directly on the hardware |
| System testing | End of the test chain, on the real overall system |
In EMC testing, the software does not have to work properly yet, it is enough to wiggle the pins to prove the electronics development. Software and hardware are developed in parallel until the product quality for the market is achieved.
The division of responsibility is clear. System testing is carried out by separate teams, while unit testing and regression testing are written by the development team itself.
Safety-critical systems have their own substructure
With functional safety systems, you work in a regulatory environment and the standards specify a lot. Above a certain safety level, you have to develop according to these specifications.
The V-model serves here as a document model with a specification page and a test page. A very formal approach is taken to ensure that the right tests are in place and you have to provide evidence of them.
This formalization has a consequence for the life cycle. Once a certified system is stable, you don’t want to go back to it lightly because every change triggers the regulatory effort again. One lever against this lies in the architecture: you only certify the part that is really relevant and separate it cleanly from the non-certified rest.
Agile development despite hardware dependency
Agile development has arrived in the embedded environment. Teams develop in short iterations, even if the hardware layouts naturally take longer.
Whether this works depends on the mindset and corporate culture, rather than the separation of hardware and software. If you don’t go with the flow in your head, it’s a problem everywhere. Hardware can certainly be developed in an agile way, but its cycles are slower.
Communication determines whether friction remains low. In EMC testing, the hardware people are stakeholders, and shortcuts are considered together instead of writing a complete test specification first. Instead of writing a document first, you clarify early on which deadlines are due and when, and how you will come together.
CI pipelines and updates over the air are standard
Continuous integration is standard in the embedded area, including automated testing on the hardware. Code changes are continuously integrated, and real target systems that run integration and system testing are in the pipeline.
This is not a luxury, but a necessity. Embedded software often consists of many modules and components on which several teams work, and everything has to fit together constantly. This also brings continuous delivery into the picture.
Software over the air is a big issue, especially in the automotive sector, for example when systems update themselves automatically. The risk is mitigated through redundancy: if an update fails, there is a fallback to the last working version. The fact that an update no longer works at all is more likely to indicate a hardware defect, such as broken memory.
Where embedded teams get stuck most often
Time-to-market is one of the biggest sources of friction. Complex development and the pressure of the market launch have to work together, and that doesn’t always work.
Embedded-specific is another trap: hardware whose functionality is specified according to the data sheet, but which then doesn’t quite work as intended. This can be a real pain in the project.
And, as everywhere, it’s requirements and communication. If the requirements are understood correctly, we implement the right things. The architect has to keep an eye on this at all times.
Security is addressed professionally, other qualities are often not
Security is professionally anchored in the embedded environment, also driven by regulatory requirements. There are guidelines designed to prevent a system from being attacked.
The reason for this is networking. Most embedded systems are integrated into an overall system and are therefore an open door for intrusion scenarios. This is being looked at, at least by the larger companies.
Companies have a harder time with other quality characteristics. Load scenarios such as user numbers tend to originate from the web world and are addressed differently from the outset with embedded systems. Maintainability and adaptability, on the other hand, are often not considered at all, and this is precisely what turns out to be expensive later on when device variants are added and the architecture concept is missing.
Where you can delve deeper into the embedded world
If you want to find inspiration for embedded development, the Embedded Software Engineering Congress (ESE) in Sindelfingen is the place to go. It takes place every year in December.
The program also covers testing topics. Over the course of a whole week, there are seminar days and many presentations, most of which are given by practitioners.
The character of the conference is informal. Despite the large number of participants, people come together closely, and this is precisely what makes it attractive as an introduction to the topic.
