Transition to open source test automation

Replacing a grown test automation framework means cutting out the existing XML-based test organization layer and replacing it with a generic framework, in this case the Robot Framework with a specially developed remote library interface for .NET. The process typically takes a year, requires internal persuasion with concrete evidence and clear exemption from the ongoing test procedure.

Key Takeaways

• An evolved test framework without a clear architecture becomes a Jenga tower: every change risks destabilizing the entire system until a new start is unavoidable.
• Persuasion is not achieved through abstract demands, but through concrete demonstration of the current status: six files open in parallel, hidden errors, unstable results.
• The Robot Framework only became usable through a self-built remote library interface server because there was no maintained .NET Core counterpart.
• The replacement of a production framework took a full year and was only possible because the manual test team completely took over the release cycles during the conversion phase.
• The main measurable benefits of the new system were shorter test runtimes, significantly less code per test case and the elimination of self-developed integrations such as reporting.

When a test automation framework needs to be replaced

A framework should be replaced when the manual work around it takes up more time and effort than the actual testing. This was precisely the trigger for Nikolaus Rieder, Test Automation Engineer at Schrack Secunet in the Communication Systems division. The old solution only appeared to run automatically. In reality, a person had to intervene permanently.

The previous automation consisted of writing procedures in XML files and observing the system. The device responded, but without a human component, nothing worked. Switching, rewiring, changing configurations, lots of copy-paste for every little adjustment: this manual work was part of every run.

In addition, there was a broken control problem. Checking tests, collating results and finding errors that had nothing to do with the device being tested took an enormous amount of time. Some errors even only occurred during the run and did not even appear in the test results.

How a Jenga tower is created from in-house code

Such frameworks rarely fail due to a single error, but rather due to unchecked growth without a target image. The old system at Schrack Secunet came from a developer who was there for a long time and built something for an acute need, without any clear specification of what it should look like. The main thing was that it worked.

A monolith grew out of this attitude. Layer upon layer was added, always following the pattern “I have to build this too”. Nikolaus describes the result as a Jenga tower: “If you pull out a piece, you have to hope that the tower stays standing.

The pattern is widespread. An Excel spreadsheet becomes too small, it becomes a database, at some point all the knowledge is in there and nobody can maintain it anymore. Over the course of 20 or 30 years, a system solidifies in which hardly anything can be moved.

Part of the problem is psychological. Anyone who starts a project is driven by the learning experience and is constantly adding something of their own. The counter-impulse is often missing: pause for a moment and ask whether it doesn’t already exist. Behind this is also a mistrust of other people’s code, along the lines that you can’t simply adopt a library from somewhere else.

The pain must be visible, not just claimed

If you want to replace a grown monolith, you first need an agreement in the team and among the product managers, not a finished new tool. The initial work was pure persuasion, illuminated from different angles in several discussions.

The effective lever was not the demand for something new, but the demonstration of the current situation. Nikolaus specifically showed the six files that he had to view in parallel and the number of errors that only occurred during the run and never appeared in the result.

I didn’t go in there with “We need something new”, but with the current status, with what I had experienced. But you have to convey this as it is from my point of view, not as it is from theirs.

• Nikolaus Rieder

Presenting the pain instead of just claiming it in the meeting creates the space for a reorganization. In a review, you show what has been achieved. It is just as important to show where the pain actually lies. Without this release, the changeover would have ended up as a side project; the full changeover took almost a whole year.

Why the Robot Framework was chosen as the tool

The Robot Framework was chosen because it is a generic testing framework and is not tied to a single programming language. Prior to this, several options were compared and discarded.

A complete commercial product such as TestStand was tried out in the test license. It did what it was supposed to do, but was too binding: you can’t get away from it. This dependency made us cautious.

A unit testing framework was also rebuilt and misused. Conceptually, this did not fit. Running unit tests at system and integration level that actually communicate with a number of devices instead of checking program code felt wrong.

The decisive factor was that the Robot Framework potentially works with different languages. This “potentially” was enough to test it seriously. The attempt to convert the existing C# code to Python was quickly abandoned because it would have meant just another layer of maintenance.

Instead, Nikolaus reverted to the nRobot server, a public domain project that was no longer maintained, and rebuilt it himself. He describes this decision as the smartest in the whole context. Relying on something else coming from outside would not have worked.

Refactoring means cutting out the XML

Porting was mainly about removing the XML test organization layer, not throwing away all the old code. Much of the old framework remained useful: communication interfaces to servers, class mappings of protocols, the basic classes of the libraries.

The code base passed two core projects, one for each trade, with project references across the board and a fully compiled console application. The procedure was similar to removing the Jenga tower from below: Finding the basic classes block by block and transferring them to the new project, sorting out what was needed and what was just hanging on the XML rail.

The XML was almost everywhere. Nikolaus compares the work to a surgical procedure: cutting out every reference to the XML from many interfaces and replacing it with something new. He describes the XML as a tumor that had spread through the entire code base.

If you build the target structure in parallel, it is easier to refactor. Because Nikolaus was building the remote library interface at the same time, he knew the keyword hierarchy and what the function calls would look like later. His colleague had to read in longer for the same step.

Parallel operation instead of big bang: securing the migration

During the changeover, the ongoing release operation must be covered differently, otherwise it will block the migration. At Schrack Secunet, the division into two trades helped. The fire department is the stronger business branch, while the communications department was able to take a break for the release test cycles.

The manual testing team took over during this time. It carried out full testing for the releases concerned, which extended the testing time there and caused stress, but freed up Nikolaus as the sole developer. This allowed him to concentrate fully on the remote library interface.

The sequence was determined by the structure: the remote library interface came first, then one C# assembly after the other was brought in. The colleague ported the fire part in his free time between release cycles because Nikolaus was not fully familiar with the product-specific code.

If this had been done in parallel with the full release operation, it would not have gone so smoothly. This gives a clear indication for our own migrations: If possible, release a responsible person from the day-to-day business instead of expecting the changeover on the side.

In the end, the year was not quite enough. A new boost system was added, a new protocol layer had to be implemented in the middle of the new framework, and the necessary integration testing took longer than expected. In addition, the physical test environment had to be rebuilt. This overlap caused delays and overload.

Why success cannot be expressed in a single key figure

Nikolaus was unable to quantify the benefits of the replacement in terms of classic bug metrics because the old framework had not recorded problems at all. Anything that was not manually entered as a bug did not appear anywhere. There were simply no clean before and after error statistics.

Instead, tangible comparisons were used as an argument. For the same routine, the team showed how many lines of code the old test case needed and how few the new one needed. Then there was the runtime: runs that took 20 or 30 minutes in the old system ran measurably faster in the new one.

The strongest argument was the elimination of development effort. Integrations such as the connection to X-Ray for Jira are built into the Robot Framework. What previously had to be built in-house is now available and frees up time.

Built-in reporting was a separate point of sale. In the old system, reporting meant a lot of manual work. In the new one, all you have to do is look at the report and there’s a clear sentence about what the test does.

An old framework needs to be consistently turned off

If the new solution is viable, the old framework should not continue to be used as a safety net. Nikolaus never actively used it again after the successful changeover. It only exists as a reference to see how an old test case was designed, not to execute tests with it.

The transition was less smooth for his colleague. Because some things that had not yet been ported were running in the release cycle, there was a back and forth between old and new, with the corresponding friction. If you’re a freelancer, you turn off the old faster than someone who is involved in day-to-day business.

The most important lesson from the year is patience. Making something new out of something old takes time, and stress leads to messing up again in the new system and poor implementation. Calm refactoring protects the quality of the new solution. Replacing a monolith not only rebuilds a framework, but also trains your own handling of lengthy processes.