Balder is an open source Python test framework under the MIT license that separates abstract test logic from the implementation. The same end-to-end tests run on several product variants, say a chest strap sensor and a watch, without duplicated test code. If a protocol or sensor changes, only the implementation is swapped and the test stays untouched.
Key Takeaways
- Established test frameworks reach their limits with product variants, because the core functions stay the same while each new variant adds or drops a protocol or an interface.
- In the Python test framework Balder, the abstract test logic is called a scenario and the implementation a setup. A matching algorithm works out which setup fits which scenario.
- Balder is not built on top of Pytest, because Pytest has no such matching. The framework was therefore written from scratch and has no dependencies.
- The open source test framework grows through BalderHub packages, modeled on Docker Hub, with ready-made tests for Selenium, page objects, CRUD data models, REST and the Django admin interface.
- Max Stahlschmidt has built Balder largely on his own so far and is looking for contributors; around 30 BalderHub projects are on GitHub, many still in beta.
A Python Test Framework for the Product Variant Problem
Balder is a Python test framework that Max Stahlschmidt built for an everyday situation in embedded testing. A manufacturer ships several product variants that share most of their functions, yet something changes from one variant to the next. A new protocol gets added, one interface disappears, a function gets optimized. The core stays the same, but the tests need attention every time.
At his earlier job, Max tried to reflect this in the test automation with established test frameworks, building test elements so they could serve several variants. That worked up to a point, then stopped. Balder grew out of that limit.
A test in Balder describes what it checks, but not what it checks with. The technology underneath, the protocol, the concrete device, sits in a swappable layer. If the protocol changes, you swap only that layer, and the test stays.
How Does Balder Separate Test Logic From Implementation?
Balder separates the abstract test logic, the scenarios, from the implementation, the setups, and brings the two together at runtime with its own matching algorithm. A scenario describes what a test needs: a device that sends out a heart pulse, say, a second device that can read it, and a connection between them. The test itself sits underneath the scenario and calls only abstract functions. A setup describes what is actually there and assigns an implementation to each feature of each device.
“In scenarios, you define what you need to run this test. In the setup, I write down what I have. And then later Balder automatically does the mapping.”
(Max Stahlschmidt)
A simple case: a simulation device sends out a signal, a reader picks it up, and the test checks whether the difference between the two values is within tolerance. At this point, nothing says which technology is involved or what the device even is, yet the test exists. Only the implementation fills the abstract functions for the concrete use case. The same test runs everywhere and still allows customization at any point.
With several products, you stay in the same project and call different variants; big projects can move tests into separate packages. Either way there is no duplicated test code, not with three products and not with more.
Around 100 End-to-End Tests From Chest Strap to Watch, Two Features Swapped
The proof came from a pilot project in which Max and his colleagues ordered sports heart rate sensors from different manufacturers, the chest straps people wear for jogging, and tested them end to end. For the first sensor, they wrote the tests from scratch. That device could do Bluetooth and no other protocol. The second sensor also supported ANT+, a protocol common in the sports world. With Balder, the team ported the existing tests, such as those for measurement accuracy, to ANT+ and kept the whole set, adapting only what had changed in the feature set.
The harder case was the watch. A chest strap measures the electrical impulse through electrodes, a sports watch measures optically. Until then, around 100 end-to-end tests had simulated a pulse as an electrical signal through a function generator. In a classic setup, a product that does functionally the same thing would have forced a rewrite of all 100 tests. In Balder, it came down to two features.
| Product | What is different | What was touched in Balder |
|---|---|---|
| Chest strap 1 | Bluetooth, small feature set | tests written from scratch |
| Chest strap 2 | ANT+ in addition | tests ported to ANT+, changes only in the altered feature set |
| Sports watch | optical instead of electrical measurement | two features reimplemented, around 100 tests carried over unchanged |
The first feature is, roughly, “simulate the heart for me”. Its old implementation produced an electrical signal through a function generator; the new one simulates light. The second feature covers the automatic switching on and off when you put the device on and take it off. With the function generator, switching off the signal is enough; with optical measurement, the test has to simulate ambient light so the device detects that it has come off. Max’s team replaced these two implementations, and the rest of the tests ran through on the new device.
Why Was Balder Not Built on Top of Pytest?
Because Pytest has no concept of this matching between scenario and setup, and bending it that far would have meant repurposing too much of it. Building on Pytest was on the table at the start, but in Max’s assessment the result would have been confusing and harder to work with, and it would not have fit the Pytest plugin ecosystem anyway. So Balder is a Python test framework with no dependencies, built from scratch and usable on its own.
The flip side is that test cases live inside the Balder ecosystem. Since a test case only calls abstract things that Balder maps together at runtime, porting the cases to other tools is hard. That was never the goal. Whoever uses Balder uses the system as a whole.
Externally, the architecture is open. Balder is MIT-licensed and has a plugin engine that accepts any extension. For reporting, there is currently a JUnit plugin; anything that works with JUnit files works with that. More integrations are in the pipeline; until they land, other tools need their own plugin. The plugin engine is not yet as built out as Pytest’s, Max says, but the most important hooks already connect easily.
Does Balder Work for Web Testing as Well as Embedded Systems?
Yes. The framework is open in terms of domain, even though it started in embedded; the push toward the web came from customers of Max’s company. In theory it had to work; in practice, the packages that carry the actual testing functionality were missing, so the past few months went mainly into the web.
For browsers, there is a Selenium package, an abstraction layer above the browser control; Playwright would be a conceivable swap, but currently there is only Selenium. An HTML package allows page objects, where you name a div element with its selector and target it without touching Selenium again. Max sees the biggest benefit in the CRUD package (Create, Retrieve, Update, Delete) together with the Data package. There you define data models, say a user with first and last name, specify which fields are optional and which must be a string, and get test logic and verification included.
In practice, you create an object through the REST package, where the features for reading and writing are already finished. For your own frontend, you write a feature that reads the data back from the page object. Balder then runs through the tests, checking for instance whether required fields produce the right error messages. All that remains is the interface specific to your own application. Almost everything Max has seen in web applications so far fits this pattern. On the web, he considers the approach just as useful as in embedded, where the variant problem is simply more pressing.
BalderHub: Ready-Made Test Packages Modeled on Docker Hub
BalderHub packages are prebuilt tests and features that you pull into Balder and only configure. The idea follows Docker Hub: what others have already built, you add to your project. Around 30 such projects sit on GitHub next to Balder itself.
One example is the package for the Django admin interface. Because that interface comes prebuilt, ready-made features exist for it, and testing it means configuring what should happen. While building this package, Max used AI, specifically to dig out the many selectors. The Balder engine itself, by contrast, is still handcrafted, without AI agents.
On the embedded side, there are packages for function generators and oscilloscopes. Many BalderHub packages are still in beta, though, and parts that belong on top are missing.
A Framework Looking for Co-Developers
Balder has been officially released for less than a year; development goes back much further, and most of it Max did alone. A few people use the open source test framework actively, but hardly anyone has contributed to the core or to Hub packages so far. His ask to people who enjoy open source: build Hub projects or work directly on the core. Both are open on GitHub.
Max’s order for the next months starts with finishing the beta packages. For embedded, a package for logic analyzers comes next, so that Balder can send out arbitrary waveforms and sequences and read them back in. For the web, he plans finished packages for widely used frameworks such as Django Admin, where everything is predefined and a test package built once can grow alongside. Anyone who builds on it plugs right in and skips writing their own. The list behind that is longer than one person can work through, as Max says himself.
