Dynamic analysis for embedded systems

Embedded testing refers to the testing of software on real hardware under real-time conditions, because simulations cannot fully reflect the physical and temporal properties of embedded systems. The gold standard for this is Embedded Trace: a hardware interface in the microcontroller that sends the control flow to the outside without influencing the application, thus enabling code coverage measurement directly in integration testing for the first time.

Key Takeaways

• Embedded Trace reads the control flow of a microcontroller via dedicated hardware interfaces without changing the execution time of the application, which classic instrumentation cannot do in principle.
• Code coverage measurement in integration testing on real hardware is now technically possible and solves the central problem that previously only unit tests on the PC or on the target could be measured in isolation.
• Instrumented code inflates memory requirements to such an extent that safety-critical embedded systems simply no longer have room for it, because on average every fifth instruction is a decision that must be followed by a counter.
• Embedded testers need to have knowledge of both electrical engineering and computer science because a bit is not set if the circuit behind it is not correctly closed or opened.
• Safety standards such as DO-178C already allow coverage verification through integration testing, but in practice they are still predominantly fulfilled at unit level, although this is more expensive and less meaningful.

What makes embedded testing more difficult than testing for web applications

Embedded testing differs from classic software testing primarily due to the scarcity of resources, real-time behavior and close integration with electronics. Anyone testing embedded systems cannot avoid the basics of electrical engineering.

The memory is small, space is limited and the time requirements are strict. A bit is only set in the memory if the circuit behind it is also correct. If you can’t read a circuit diagram and don’t know Ohm’s law, you can’t do meaningful testing at this level.

Then there is the profile of the application. An embedded system works in real time. If you ignore this real-time behavior during testing, you may not be testing the system that will actually run later.

The job profile reflects this requirement. Many embedded developers come from electrical engineering, not from pure computer science. Martin Heininger studied electrical engineering himself. Both worlds have to come together, and that’s exactly what you need to be able to do.

Why the hardware is the truth

In the safety-critical embedded area, the hardware is the only reliable test environment. A simulation is not sufficient for acceptance because it is almost impossible to prove its correctness.

Testing can certainly be carried out on the PC, especially at unit level and under certain conditions. However, the further you get towards system and acceptance testing, the more there is no way around real electronics.

The reason for this lies in the question that every safety verification raises: Can you prove that the simulation reflects the truth? In practice, this is rarely possible. So you test on the physical system.

Hardware reality has also become more complex. In the past, there was a CPU with external memory, clear models and predictable runtimes. Software needed a certain number of CPU clocks and was ready at a defined time.

In today’s multicore systems with caches, processes displace each other from the memory. The determinism is gone or can only be restored with great effort. Precise models no longer exist for many modern SoC systems. What remains is trial and error on the real system, combined with the ability to look inside.

Where embedded testing differs from IT testing

Three areas clearly separate embedded testing from classic IT testing: volume testing, code coverage requirements and observability.

The volume test with huge databases and real personal data does not exist in the embedded area. What is considered load testing there is more like stress testing on a bus, such as CAN or Ethernet, with comparatively little data. For an IT tester, these volumes of data would hardly be worth mentioning.

The situation is reversed when it comes to code coverage. In the IT environment, 100 percent coverage, every line or even every condition, is neither common nor sensible given the sheer size. In the security environment, the opposite is true: if there is an error in the code, a real problem arises. So everything has to be tested.

Security also follows a different pattern. IT has a head start here because it has longer experience with appropriate testing and protection concepts. In the embedded sector, the topic is only just beginning. A common strategy is: if there is an interface to the outside, the larger system on the other side must protect the small embedded system.

Why instrumentation reaches its limits in integration testing

Instrumentation works great for isolated unit testing, but becomes a problem for system and integration testing. It changes the system that you actually want to test.

The mechanism: Approximately every fifth instruction is a decision that must be logged. Behind every decision is a counter that is read, incremented and written back with every run. This greatly inflates the code.

This results in two unpleasant effects. Firstly, the instrumented code often no longer fits into the limited memory. Secondly, the execution time changes. A read-modify-write in the cache is fast, in the external memory it quickly takes a factor of 100 longer.

In a system with guaranteed response times or reactions to interrupts, this changes the behavior. You end up testing the instrumented system, which has only a limited connection with the actual application.

In aviation, this issue is so critical that some of the instrumentation remains in the release system so that exactly the tested code is delivered. Other areas test once with and once without instrumentation and hope that both variants behave in the same way.

Embedded Trace: Observe without disturbing the system

Embedded Trace provides insight into the running system without the application noticing anything. A hardware unit on the CPU transmits what the CPU is doing to the outside world without changing the application.

This non-reactive observation avoids the central problems of instrumentation. No bloated code, no distorted runtime. Alexander Weiss describes this as the gold standard of observation, albeit with clear prerequisites.

There is hardware on the CPU that transmits what the CPU is doing to the outside world without the application being aware of it.

Alexander Weiss

The most important requirement is of a physical nature: although the trace interface is located on almost every modern microcontroller, it must also be accessible on the board. In early hardware versions, the trace points are often collected in different places on the board until the appropriate connector is available in later redesigns.

The practical advice from this: write it into the specifications early on that the trace point must be provided. It is not needed for productive operation, but for debugging and testing, and that is exactly where it will otherwise be missing.

How trace data is evaluated

Trace data cannot be evaluated in software in real time, but requires dedicated hardware. Even the fastest microcontroller produces data volumes that a pure software decoder could not calculate live.

There are several relevant protocols for transport. Parallel output via many pins is possible, but unsightly due to the pin requirements for bandwidths in the gigabit range. High-speed serial interfaces such as Aurora output several gigabits via one lane, optionally via several lanes.

A third variant uses a system interface such as PCI Express. This is elegant and widely available, but slightly intrusive: as soon as the trace unit grabs bandwidth on the PCI Express, a backlash occurs again. A dedicated serial trace interface remains technically the cleanest solution, but is not always available.

The evaluation is performed by an FPGA, a freely programmable large chip. It records the individual trace snippets in massive parallel and calculates the protocol. There is a separate hardware counter for each instruction address for code coverage. After the measurement, the sum of all counters shows how often which point of the control flow was traversed.

The trace protocol is highly compressed. The specific address jump is not output, but only whether a direct jump was taken or not. Without the application binary, this data stream cannot be interpreted, which also alleviates security concerns. If you want to additionally secure the trace interface, you can lock it permanently using Fuse.

Code coverage in integration testing instead of unit testing

Embedded Trace makes it possible to measure code coverage on the real target in system and integration testing. Until now, coverage was practically limited to the unit test due to the instrumentation.

The original idea behind coverage measurement originated in aviation around 1992. Requirements and tests share a weakness: you never know exactly when they are complete. Code coverage, on the other hand, is mathematics based on the source code; at 100 percent, the statement is unambiguous.

Coverage was intended as a tool for detecting missing requirements and missing tests by setting coverage against both. However, this failed at unit level because the requirements are missing there. It is precisely this gap that the measurement closes in integration testing.

At this level, measurements can even be carried out during real operation, theoretically in a test flight, practically in the laboratory on the individual control unit. The hardware delivers the truth and the code that has been run becomes visible.

Martin sees the benefits above all in functional safety. Formalized unit testing in safety is very expensive today with large test teams and often provides little added value.

Integration testing before unit testing, with mandatory justification

It would make sense to give integration testing priority and only use unit testing as a supplement where coverage cannot be achieved otherwise. The justification should come before the additional unit tester, not after.

Aviation standards such as DO-178C leave both paths open today. You can achieve the required coverage via integration testing or via unit testing. This freedom was also created in the knowledge of the technical possibilities at the time.

It is important to draw the line: this is not a plea against unit testing or a return to big-bang integration. Everything that a developer sensibly does in terms of unit testing should remain in place and belongs back in the hands of the developer. What is being criticized is the excessive formalism that ties up a lot of money in safety.

Standards always reflect the state of the art at the time of their creation. In the software sector, many things develop quickly, which is why a standard may already be lagging behind when it is published. For a technology such as embedded trace to get its chance, standards must leave room for new procedures.