CI/CD pipelines for embedded software create a fault-tolerant development environment in which changes are automatically tested before they flow into the common code base. The core: unit tests span a safety net that makes regressions visible. Supplemented by automatic mock generation, a standardized build system and integrated documentation, R-SPICE-compliant quality can be systematically ensured.
Key Takeaways
- Unit tests are not a proof of quality for the current code, but a safety net for all future changes: The more tests there are, the more peace of mind a team can have in developing its software.
- If you fix a bug, you first write a tester that reproduces this bug, making the test network tighter instead of just patching the bug.
- Marquardt relies on its own open source platform called SPLE, which combines CMake, Google Test, KConfig and the self-developed Automocker Hammocking to create a consistent embedded C development environment.
- Sphinx-based documentation generation makes it possible to keep requirements, test specifications and test results in the same source code repository and generate R-SPICE-compliant component reports from them.
Why embedded teams fail at software quality when they come from hardware
Embedded software is often created in tinkering mode, and this is exactly what becomes a problem later on. Those who come from hardware development rarely start with a well thought-out build environment. He hacks together a first stand, the LED flashes, and that is the moment of joy.
Karsten Günther describes this appeal as the core of programming: you build something, and in the end it does something. In the embedded sector, you can do this quickly. As long as the product remains small, this mode works.
The break comes with complexity. As soon as a product is sold and several customers have their own requirements, hacking is no longer enough. Automated testing is then needed to maintain consistent quality. This hurdle affects many companies with a strong hardware heritage in equal measure.
At Marquardt, a manufacturer of battery management systems, touch panels, LED strips and control elements, the majority of software developers originally come from hardware. This results in a backlog in software development that cannot be closed by individual tools, but by a different way of working.
What CI/CD should really achieve in the embedded context
CI/CD is the answer to automation and repeatable testing. If you want to automate and repeat regression testing, there is no way around continuous integration.
Continuous integration means importing changes into a mainline several times a day and making them available to everyone. Several people work on a code base, each change is tested and the quality remains at the same level.
The actual goal lies deeper than the technology. Continuous integration is intended to create an error-tolerant environment. If someone makes a mistake, this one person should receive the feedback, not the entire organization.
Karsten talks about an early job where he was told: This is your code, you’re responsible for it, but don’t change anything, you can break things. For a software developer, that’s a contradiction. His job is to change software. If change is dangerous, the safety net is missing.
Platform engineering: developers build products, not build systems
No developer wants to operate a build system. Developers enjoy implementing functionality, not maintaining tools. This is exactly where platform engineering comes in.
Marquardt is building a standardized platform, known internally as SPLE, for software product line engineering. The platform provides developers with everything they need to develop their product, make it configurable and create variants.
Variant creation is not a secondary aspect here, but the business model. Marquardt sells products to customers who stand between the manufacturer and the end user and want individual customization. An LED in a car, a washing machine button, a battery management system: every customer wants their own customization.
The task of the platform is to empower the developer. They should be able to configure their software, test this configuration and integrate their changes without breaking anything.
Test-Driven Development provides a safety net, not proof
A test does not prove that your current code works. This reversal is the core of Test-Driven Development. If you solve a simple task, such as adding two numbers, you can write working code without testing.
The value only becomes apparent with extensions. If new requirements are added, nobody knows all the features that have already been implemented in detail. The probability of destroying something old when making changes is real.
This is exactly where the test network comes in. Existing tests create a safety net. If the developer makes a mistake with a new requirement, a test fails and he immediately sees what he has broken. The more tests, the more relaxed the work.
You first write a test to reproduce the bug. And that’s not a waste, because this test only makes your test network tighter.
- Karsten Günther
Karsten describes this as a gift. Even when fixing bugs, you first write a test that reproduces the bug. This test remains and narrows the net permanently. Over the years, a component grows to thousands of unit testers that run with every change and provide quick feedback. No software is bug-free, but the narrower the network, the lower the probability of old features being destroyed unnoticed.
The technical cornerstones of an embedded CI platform
Marquardt’s SPLE platform is based on several specific tools, most of which are open source. They interlock instead of existing side by side.
| Cornerstones | Function |
|---|---|
| CMake | build system, mature and powerful, maintained for many years |
| Git with branch protection | Trunk-based development, changes must go through a gate |
| Google Test | C++-based test framework as test runner |
| Hammocking | self-developed Python package for automatic mockup generation |
| Sphinx | Documentation generator for reports and requirements tracing |
CMake is the foundation for the build environment. Marquardt has extended it so that it supports software product line engineering and thus variants, which CMake does not provide out of the box. It also includes K-Config, some PowerShell and Python.
Google Test brings the full power of object-oriented C++ to testing. For developers from the pure C world, getting started is a technical hurdle because Google Test is C++. Once you have arrived, you can do things that are otherwise more familiar from Java or Python.
Hammocking: Generate mocks automatically instead of cutting them by hand
C development lacks the mocking that object-oriented testing frameworks often provide. In unit testing, the element to be tested must be cut free from its external interfaces. These dependencies need to be removed.
Marquardt has built his own tool for this: Hammocking, a Python package that automatically generates mockups. The name is a playtest on Hammock and Mock. The return values of external interfaces can thus be used as input or output vectors in unit testing.
Hammocking is open source and available on GitHub. Marquardt uses it internally, but continues to develop it publicly. The same applies to the entire SPLE platform, which can be downloaded and installed.
Documentation belongs to the code, not in a separate tool
Reporting and verification requirements can be integrated into the same platform used for testing. Marquardt uses Sphinx, the documentation tool from the Python world and the basis of many read-the-docs pages.
The platform covers requirements tracing, test spec generation and automatic test reports via Sphinx extensions. The documentation is ASPICE-compliant for unit construction because users in the company have requested requirements tracing and the writing of software requirements.
The advantage lies in the storage location. All documentation is stored in the source code repository, directly with the code. A report can be generated for each component, bringing together the design spec, test specification and test results.
The test specification is generated from the Google test suites and the test results are included directly. This means that the path from the requirement to the test result is ASPICE-compliant, with no breaks between tools.
Where an embedded platform must develop to
The next stage is in-house distribution. The aim over the next two years is to move as many projects as possible that are still based on old technologies onto the SPLE platform. User acceptance is already high.
In terms of content, ASPICE goes beyond unit construction. ASPICE describes the entire V-model, and that is exactly the direction: the platform should support the V-model as far as software development needs it.
Specifically, this means hardware in the loop, software in the loop, software integration testing and system integration testing. The platform is intended to create the basis for implementing these test levels as easily as possible. In this way, a build system for unit testing gradually becomes a consistent quality environment.
