Moving from a homegrown test automation framework to Robot Framework means cutting out the old XML-based test organization layer and replacing it with a generic framework, here extended with a custom remote library interface for .NET. At Schrack Seconet the switch took about a year. It needed internal buy-in backed by concrete evidence and a developer who was freed from regular release testing.
Key Takeaways
- A test framework that grows without a clear architecture turns into a Jenga tower: every change can bring the whole thing down, until starting over is the only option left.
- Buy-in doesn’t come from asking for something new in the abstract. It comes from showing the current state: six files open side by side, hidden errors, unreliable results.
- Robot Framework only became usable after the team rebuilt a remote library interface server themselves, because there was no maintained .NET Core counterpart.
- Replacing a framework that was in production use took a full year and only worked because the manual test team took over the release cycles completely during the rebuild.
- The measurable gains were shorter test runs, far less code per test case and no more homegrown integrations such as reporting.
When a Test Automation Framework Needs Replacing
A framework is due for replacement when the manual work around it costs more than the test itself. For Nikolaus Rieder, test automation engineer in the communication systems division at Schrack Seconet, that was the reason to move his team’s test automation to Robot Framework. The old solution only looked automated. In reality, someone had to step in all the time.
The previous automation meant writing procedures in XML files and then watching the system. The device responded, but nothing ran through without a person involved. Switching settings, rewiring, changing configurations, copying and pasting for every small tweak: that manual work came with every single run.
On top of that, keeping control was a problem. Monitoring tests, collecting results and chasing errors that had nothing to do with the device under test ate up huge amounts of time. Some errors only occurred during the run and never showed up in the test results at all.
How In-House Code Turns into a Jenga Tower
Frameworks like this rarely fail because of one bug. They fail because they keep growing without a target picture. The old system at Schrack Seconet came from a long-serving developer who built something for an urgent need, with no clear idea of what it should look like in the end. As long as it worked, it was fine.
That mindset produced a monolith. Layer after layer went on top, always following the same pattern: “I need to build this as well.” Nikolaus describes the result as a Jenga tower. Pull out one piece and you have to hope the tower stays up.
The pattern is common. An Excel sheet gets too small, so it becomes a database, at some point all the knowledge lives inside it, and nobody can maintain it anymore. Over 20 or 30 years, the machinery sets so hard that hardly anything moves.
Part of the problem is psychological. Whoever starts a project is driven by what they’re learning and keeps adding their own code. What’s often missing is the counter-impulse to stop for a moment and ask whether this already exists. There’s also a distrust of other people’s code, the feeling that you can’t just pull in some library from somewhere.
How to Get the Team and Product Managers on Board
Replacing a grown monolith starts with agreement inside the team and among the product managers, not with a shiny new tool. The first stretch of work was pure persuasion, over several conversations and from different angles.
What worked was not asking for something new but showing how things stood. Nikolaus walked people through the six files he had to keep open side by side and the number of errors that only occurred during a run and never appeared in the results.
“I didn’t go in saying ‘we need something new.’ I went in with the current state, with what I’d experienced. But you have to put it across the way it looks from my side, not the way it looks from theirs.”
(Nikolaus Rieder)
Demonstrating the pain instead of just claiming it in a meeting is what buys you room for a cleanup. In a review you show what you’ve achieved. It matters just as much to show, at least once, where it actually hurts. Without that approval, the switch would have fizzled out as a side project. The full migration took almost a whole year.
Why Robot Framework Beat TestStand and a Unit Testing Framework
Robot Framework won because it is a generic test framework that doesn’t tie you to one programming language. Before that decision, the team compared several options and dropped them again.
A commercial all-in-one product, TestStand, was tried out on a trial license. It did what it promised, but the lock-in was too strong: once you’re in, you don’t get out. That dependency made the team cautious.
They also rebuilt a unit testing framework and bent it to the job. Conceptually it didn’t fit. Launching unit tests at system and integration level that really talk to a whole range of devices, rather than checking program code, felt wrong.
What tipped the balance was that Robot Framework can potentially work with several languages. That “potentially” was enough to give it a serious try.
Reusing Existing C# Code with a Remote Library Server
An attempt to convert the existing C# code to Python was abandoned quickly, because it would only have added another layer to maintain. Instead, Nikolaus went back to the nRobot server, a public domain project that was no longer maintained, and rebuilt it himself. He calls that the smartest decision in the whole effort. Waiting for someone outside to deliver an update would not have ended well.
The refactoring was mainly about removing the XML test organization layer, not about throwing away all the old code. A lot of the old framework stayed useful: communication interfaces to servers, class mappings of protocols, the base classes of the libraries.
The codebase consisted of two core projects, one per business unit, with project references running across each other and a precompiled console application. The approach was like taking the Jenga tower apart from the bottom: find the base classes one by one, move them into the new project, and sort out what was needed and what only depended on the XML layer.
The XML was everywhere. Nikolaus compares the work to surgery: at many interfaces, every reference to the XML had to be cut out and replaced with something new. He describes the XML as a tumor that had spread through the entire codebase.
Building the target structure at the same time makes refactoring easier. Because Nikolaus was setting up the remote library interface in parallel, he knew the keyword hierarchy and what the function calls would look like later. His colleague needed more time to get up to speed for the same step.
Keeping Releases Running during the Migration
While the switch is under way, regular release testing has to be covered some other way, or it will block the migration. At Schrack Seconet, the split into two business units helped. Fire safety is the stronger business, and the communication systems unit could pause its release test cycles.
During that time, the manual test team stepped in. It ran full tests for the affected releases, which made testing take longer there and caused stress, but it freed Nikolaus, the only developer on the project, to focus entirely on the remote library interface.
The order followed from the architecture. First the remote library interface had to stand, then the C# assemblies could be brought in one after another. His colleague ported the fire safety part in the gaps between release cycles, because Nikolaus didn’t know that product-specific code well enough.
Doing all this alongside full release operations would not have gone as cleanly. The takeaway for your own migration: if you can, free up one person from day-to-day work to own the switch, instead of expecting it to happen on the side.
In the end, the year wasn’t quite enough. A new product came along, a new protocol layer had to be implemented right in the middle of the new framework, and the integration test it required took longer than expected. The physical test environment was due for a rebuild as well. That overlap caused delays and overload.
Proving the Benefit without Bug Metrics
Nikolaus couldn’t quantify the benefit with classic bug metrics, because the old framework never tracked problems in the first place. Anything that wasn’t logged as a bug by hand simply didn’t show up anywhere. There was no clean before-and-after defect statistic.
So the team argued with tangible comparisons. For the same routine, they showed how many lines of code the old test case needed and how few the new one did. Then there was runtime: runs that took 20 or 30 minutes in the old system were noticeably faster in the new one.
The strongest argument was the development work that disappeared. Integrations such as the connection to X-Ray for Jira come built into Robot Framework. What used to be built in-house is now simply there, and that frees up time.
| Aspect | Old Framework | Robot Framework |
|---|---|---|
| Execution | Semi-automated, constant manual intervention | Automated |
| Code per routine | Many lines | Far fewer |
| Runtime | 20 to 30 minutes | Noticeably faster |
| Reporting | Manual and painful | Built in, readable |
| Integrations (e.g., X-Ray/Jira) | Homegrown | Built in |
The built-in reporting was a selling point of its own. In the old system, reporting meant a lot of manual work. In the new one, a look at the report is enough, and it contains a plain sentence describing what the test does.
Switch the Old Framework Off for Good
Once the new solution holds up, the old framework shouldn’t keep running as a safety net. Nikolaus never actively used it again after the switch. It exists only as a reference for looking up how an old test case was designed, not for running tests.
For his colleague, the transition was bumpier. Because parts that hadn’t been ported yet were still running in the release cycle, there was some back and forth between old and new, with the friction that comes with it. If you’ve been freed up for the migration, you switch off the old system faster than someone who is still stuck in day-to-day work.
The biggest lesson from that year is patience. Turning something old into something new takes time, and under stress you end up tangling yourself up again in the new system and implementing things badly. Staying calm during refactoring protects the quality of the new solution. Replacing a monolith doesn’t just rebuild a framework, it also trains how you handle long, drawn-out processes.
Frequently Asked Questions
How can you tell that test automation is only seemingly automated?
A reliable sign is the manual work involved. In the case described, procedures were written in XML files, and no test run would complete without switching settings, reconfiguring cables, making configuration changes, and copying and pasting. On top of that, there were errors that only occurred during the run and never showed up in the test results. The rule of thumb: Replace it as soon as the work surrounding the test takes up more effort than the test itself.
Why do development teams prefer to build their own solutions instead of using existing libraries?
There are two driving forces behind this: the learning experience, which tempts developers to constantly add their own code, and a distrust of third-party code based on the idea that you can’t just adopt a library from somewhere else. There’s no counter-impulse to pause briefly and ask whether it already exists. So, layer by layer, a monolith grows without a clear vision.
How do you convince the team and product management to replace an existing testing framework?
By presenting the current state of affairs, not by demanding something new. What proved effective was a concrete demonstration: six files that had to be monitored simultaneously, and the number of errors that occurred only during the test run. First, you need agreement within the team and among the product managers, not a ready-made tool. Without this approval, the transition will fizzle out as a side project.
Is there any reason not to choose a commercial, all-in-one product for test automation?
Vendor lock-in. A complete product like TestStand did what it was supposed to do within the test license, but it created such a strong lock-in that it became impossible to move away from it. A unit-test framework repurposed for this task was also rejected: Running unit tests at the system and integration levels (tests that, in reality, communicate with numerous devices) doesn’t make conceptual sense. The choice fell on a generic framework with no language lock-in.
Does switching to a new test framework require discarding all legacy code?
No. The XML-based test organization layer was specifically removed. Communication interfaces to servers, class mappings of protocols, and the base classes of the libraries remained useful and were migrated, piece by piece, into the new project. The sorting process focused on elements that were solely dependent on the XML layer. Since the XML was embedded almost everywhere, every reference to the interfaces had to be cut out individually.
How do you maintain release operations during a framework migration?
By having someone else take over the test cycles. The manual testing team performed comprehensive testing for the affected releases. This extended the testing time and caused stress, but it freed up the sole developer. The transition wouldn’t have gone as smoothly if it had been carried out alongside full release operations. So, if possible, assign a dedicated person from day-to-day operations to the task instead of expecting the migration to happen on the side.
How can you demonstrate the benefits of a new testing framework when there are no baseline figures?
Through tangible comparisons rather than bug metrics. The old system didn’t track issues at all: anything not manually logged as a bug didn’t show up anywhere. Therefore, the case was made using the number of lines of code for the same routine, the runtime (previously 20 to 30 minutes, afterward noticeably faster), and the elimination of custom code, such as the integration with X-Ray for Jira.
Should the old framework continue to run as a safety net after the migration?
No. After the successful migration, it was no longer actively used but now serves only as a reference to see how an old test case was designed. Where parts that hadn’t yet been ported were still running in the release cycle, there was a back-and-forth between the old and new systems, causing some friction. Those who have completed the migration are phasing out the old system more quickly.
