Dynamic analysis of embedded systems means observing software while it runs on real hardware under real-time conditions, because simulations cannot fully capture the physical and timing properties of embedded systems. The gold standard for this is embedded trace: a hardware interface in the microcontroller that streams the control flow to the outside without affecting the application. For the first time, this makes it possible to measure code coverage directly in integration testing.
Key Takeaways
- Embedded trace reads out a microcontroller’s control flow through dedicated hardware interfaces without changing the application’s execution time, something classic instrumentation fundamentally cannot do.
- Measuring code coverage in integration testing on real hardware is now technically possible. It solves the core problem that coverage could previously only be measured for unit tests in isolation, on the PC or on the target.
- Instrumented code inflates memory requirements so much that safety-critical embedded systems simply have no room left for it, because on average every fifth instruction is a decision that needs its own counter.
- Embedded testers need both electrical engineering and computer science skills, because a bit won’t be set if the circuit behind it isn’t properly closed or opened.
- Safety standards such as DO-178C already accept coverage evidence from integration testing, yet in practice most teams still meet it at unit level, even though that costs more and tells you less.
What Makes Embedded Testing Harder Than Testing Web Applications
Embedded testing differs from classic software testing mainly in three ways: scarce resources, real-time behavior and a tight coupling with the electronics. That is why dynamic analysis on the real hardware matters so much here, and why anyone testing embedded systems needs to know the basics of electrical engineering.
Memory is small, space is tight and the timing requirements are strict. A bit only gets set in memory if the circuit behind it is right. If you can’t read a circuit diagram and don’t know Ohm’s law, you can’t test meaningfully at this level.
Then there’s the nature of the application. An embedded system runs in real time. If you ignore that real-time behavior in testing, you may end up testing something other than the system that will actually run later.
The profession reflects this. Many embedded developers come from electrical engineering rather than pure computer science. Martin Heininger studied electrical engineering himself. Both worlds have to come together, and you have to want exactly that from the start.
Why the Hardware Is the Ground Truth
In safety-critical embedded work, the hardware is the only reliable test environment. A simulation isn’t enough for acceptance, because its correctness is almost impossible to prove.
You can certainly test on the PC, especially at unit level and under certain conditions. But the closer you get to system and acceptance testing, the less you can avoid the real electronics.
The reason is the question every safety case raises: can you prove that the simulation reflects reality? In practice, you rarely can. So you test on the physical system.
On top of that, hardware has become more complex. There used to be one CPU with external memory, clear models and predictable execution times. A piece of software needed a certain number of CPU cycles and was done at a defined point in time.
In modern multicore systems with caches, processes push each other out of memory. Determinism is gone, or can only be restored with a lot of effort. For many modern SoCs, accurate models simply don’t exist anymore. What’s left is trying things out on the real system, combined with the ability to look inside it.
Where Embedded Testing Differs from IT Testing
Three areas clearly set embedded testing apart from classic IT testing: volume testing, code coverage requirements and observability.
Volume testing with huge databases and real personal data doesn’t exist in that form in the embedded world. What counts as load testing there is closer to a stress test on a bus such as CAN or Ethernet, with comparatively little data. An IT tester would barely notice those data volumes.
With code coverage, it’s the other way around. In IT, 100 percent coverage of every line or even every condition is neither common nor sensible, given the sheer size of the systems. In a safety-critical environment, the opposite applies: a defect in the code causes a real problem. So everything has to be tested.
Security follows a different pattern, too. Here IT is ahead, because it has more experience with the relevant tests and protection concepts. In the embedded world, the topic is only just getting started. A common strategy: if there is an interface to the outside, the larger system on the other side has to protect the small embedded system.
Why Instrumentation Hits Its Limits in Integration Testing
Instrumentation works very well for isolated unit tests, but it becomes a problem in system and integration testing. It changes the very system you want to test.
Here’s how it works: roughly every fifth instruction is a decision that has to be recorded. Behind each decision sits a counter that is read, incremented and written back on every pass. That bloats the code considerably.
This has two unpleasant consequences. First, the instrumented code often no longer fits into the limited memory. Second, the execution time changes. A read-modify-write in the cache is fast, but in external memory it can easily take 100 times longer.
In a system with guaranteed response times or interrupt reactions, that tips the behavior over. In the end, you are testing the instrumented system, which has only a loose connection to the actual application.
In aviation, this is so critical that the instrumentation sometimes stays in the release build, so that exactly the tested code gets shipped. Other domains test once with and once without instrumentation and hope that both variants behave the same.
Embedded Trace: Dynamic Analysis without Disturbing the System
Embedded trace gives you a view into the running system without the application noticing. A hardware unit attached to the CPU broadcasts what the CPU is doing, without changing the application at all.
This non-intrusive observation avoids the core problems of instrumentation. No bloated code, no distorted timing. Alexander Weiss calls it the gold standard of observation, though with clear prerequisites.
“There’s hardware on the CPU that broadcasts what the CPU is doing, without the application noticing.”
(Alexander Weiss)
The most important prerequisite is physical. Almost every modern microcontroller has a trace interface, but it also has to be accessible on the board. In early hardware revisions, the trace signals are often picked up from various spots on the board, until a later redesign finally includes the right connector.
The practical advice: put it in the requirements specification early that the trace port must be provided. It isn’t needed in production, but it is needed for debugging and testing, and that is exactly where it will otherwise be missing.
How Trace Data Is Evaluated
Trace data can’t be evaluated in software in real time. It needs dedicated hardware. Even the fastest microcontroller produces more data than a pure software decoder could process live.
Several protocols are relevant for getting the data out. Parallel output over many pins is possible, but awkward, because gigabit bandwidths need a lot of pins. High-speed serial interfaces such as Aurora push out several gigabits per lane, optionally over multiple lanes.
A third option uses a system interface such as PCI Express. That is elegant and widely available, but it has a catch: as soon as the trace unit takes bandwidth on PCI Express, it becomes intrusive again. A dedicated serial trace interface remains the cleanest technical solution, but it isn’t always available.
The evaluation is done by an FPGA, a large, freely programmable chip. It takes in the individual trace fragments massively in parallel and decodes the protocol. For code coverage, it holds a separate hardware counter for each instruction address. After the measurement, the counters add up to how often each point in the control flow was executed.
The trace protocol is heavily compressed. It doesn’t output the actual jump address, only whether a direct branch was taken or not. Without the application binary, the data stream can’t be interpreted, which also eases security concerns. If you want to lock down the trace interface on top of that, you can disable it permanently with a fuse.
Code Coverage in Integration Testing Instead of Unit Testing
Embedded trace makes possible what long wasn’t: measuring code coverage on the real target during system and integration testing. Until now, instrumentation effectively limited coverage to unit testing.
The original idea behind coverage measurement comes from aviation, around 1992. Requirements and tests share a weakness: you never know exactly when they are complete. Code coverage, by contrast, is mathematics based on the source code. At 100 percent, the statement is unambiguous.
Coverage was meant as a tool for finding missing requirements and missing tests by checking coverage against both. At unit level, that didn’t work, because there are no requirements at that level. Measuring in integration testing closes exactly that gap.
At this level, you can in principle even measure during real operation: in theory during a test flight, in practice in the lab on a single control unit. The hardware tells the truth, and the code that actually ran becomes visible.
Martin sees the main benefit in functional safety. Formal unit testing in safety projects is very expensive today, with large test teams, and often adds little value.
Integration Tests First, Unit Tests Only with Justification
It would make sense to give integration testing priority and add unit tests only where coverage can’t be reached any other way. The justification should come before the additional unit test, not after.
Aviation standards such as DO-178C leave both routes open today. You can achieve the required coverage through integration tests or through unit tests. That freedom also reflected what was technically possible at the time.
The distinction matters: this is not an argument against unit testing, and not a return to big-bang integration. Everything a developer sensibly does in unit testing should stay, and it belongs back in the developer’s hands. The criticism is aimed at the excessive formalism that ties up a lot of money in safety.
Standards always reflect the state of the art at the time they were written. Software moves fast, so a standard can already lag behind when it comes out. For a technology like embedded trace to get its chance, standards need to leave room for new methods.
Frequently Asked Questions
What prior knowledge does someone need for embedded software testing?
Embedded testers need knowledge of both electrical engineering and computer science. A bit is set in memory only if the underlying circuit is correctly closed or open. Anyone who cannot read a circuit diagram and does not know Ohm’s law cannot test effectively at this level. Consequently, many embedded developers come from an electrical engineering background rather than pure computer science.
Is a simulation sufficient as a test environment for safety-critical embedded software?
No. Simulation is not sufficient for acceptance testing because its correctness is virtually impossible to prove. Any safety verification raises the question of whether the simulation accurately reflects reality, and in practice, this is rarely possible. At the unit level, testing on a PC is certainly feasible. The closer you get to system and acceptance testing, the less you can avoid using actual electronics.
Why is the time behavior of today’s microcontrollers harder to predict than in the past?
In the past, there was a CPU with external memory, clear models, and predictable runtimes: A piece of software required a specific number of CPU cycles and was finished at a defined point in time. In multicore systems with caches, processes displace one another from memory; determinism is lost or can only be restored with great effort. For many SoC systems, precise models no longer exist.
Why is 100 percent code coverage considered a goal in the safety domain but not in IT testing?
Because the consequences of an error vary in severity. In the safety domain, an error in the code leads to a real-world problem, so everything must be tested. In the IT environment, 100 percent coverage of every line or every condition is neither common nor practical given the sheer size of the systems. In volume testing, the ratio is reversed: embedded load testing moves comparatively little data.
To what extent does instrumentation change the behavior of an embedded system?
Significantly, and in two ways. Roughly one in five instructions is a decision backed by a counter that is read, incremented, and written back on every pass. The code expands so much that it often no longer fits into the limited memory. In addition, runtime increases: a read-modify-write operation in external memory can easily take 100 times longer than in the cache.
What needs to be planned early on to ensure that Embedded Trace can even be used later?
Physical access. The trace interface is present on almost every modern microcontroller, but it must also be accessible on the circuit board. In early hardware versions, trace points are often pieced together from various locations on the board until a later redesign provides the appropriate connector. This note should be included in the specifications: The location is not needed for production operation, but rather for debugging and testing.
Why can’t trace data simply be analyzed using software?
Even the fastest microcontroller generates data volumes that a pure software decoder cannot process in real time. The analysis is therefore handled by an FPGA, which captures the trace snippets in massively parallel fashion and calculates the protocol. For code coverage, there is a separate hardware counter for each instruction address. The sum of all counters ultimately shows how often each part of the control flow was executed.
Do coverage measurements in integration testing make unit tests unnecessary?
No. It makes sense to give priority to integration testing and use unit tests as a supplement where coverage cannot be achieved otherwise, though the rationale should come before the additional unit test. Any unit tests that a developer reasonably writes remain in place and are their responsibility. What is criticized is the excessive formalism that ties up a lot of money in safety.
