Automatic test case selection for regression tests - Silke Reimer 

 June 20, 2023

You are currently viewing a placeholder content from Podigee. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.

More Information

"On the Java side, we were previously at a runtime of 2.5 hours (...) In many cases, we can now actually give feedback to the developers after 10-15 minutes: Yes, it was good - or not" - Silke Reimer

Silke had a typical problem with regression tests in her company: they ran for a very, very long time. The framework conditions were difficult: The regression tests check a complex test object that is developed in different programming languages. Together with the TU Munich, she and her team were able to solve the problem - through an automated test selection across the C++ and Java parts. In the interview, we talked about the concrete implementation, different concepts for test selection and the impressive numbers about the time savings achieved.

Silke Reimer came into contact with the concept of Test Driven Development more than 20 years ago at her first employer. Since then, the topic of test automation and software quality assurance in general has never really left her. In the meantime, she is head of quality assurance for one of the two business units at IVU Traffic Technologies AG. An important part of her job involves improvements in the area of test automation at all levels. Prior to her current position, she also dedicated many years to this topic as a test architect. It is especially important to her that test automation is not the goal but one of several measures to give the developers a quick feedback about the quality of the code.

Highlights in this episode:

  • We talk about test case selection for regression tests
  • The solution was developed together with the Technical University of Munich to reduce throughput times
  • Silke works at IVU and develops software for transportation companies
  • The test suite comprises around 25,000 test cases and took up to seven hours without selection
  • Selection can save up to 50-60% of test cases
  • The solution analyzes which files are opened during test runs in order to select the necessary test cases
  • A reduction from hours to minutes was achieved in some runs
  • Developers receive significantly faster feedback
  • The system has been running for around one and a half to two years
  • Es gab keine Situationen, in denen wichtige Fehler durch das System übersehen wurden

 

Contact Silke:

 

How to turn hours into minutes

TLDR: Silke Reimer shares exciting insights into the world of automatic test case selection. In collaboration with the Technical University of Munich, an innovative solution was developed to significantly reduce the throughput times of regression tests in a complex software environment.

The leap into a new era of test automation

Silke talks about challenges and solutions in a scenario that many companies are very familiar with: long throughput times for regression tests due to a huge code memory and various programming languages. The collaboration between IVU and the Technical University of Munich led to the development of an innovative approach that makes it possible to execute only the necessary test cases and thus achieve massive time savings.

The challenge: A mountain of code and waiting time

Silke gave an overview of the dimensions of the problem. At IVU, a company that develops software for transportation companies, the test suite alone comprises around 25,000 cases. A complete execution of all tests took up to seven hours - an unbearable waiting time for developers after each code check-in. In addition to the complexity caused by different programming languages such as C++ and Java, there was also the maintenance of numerous branches. This setting led to an enormous strain on resources and a delayed feedback loop.

The solution: Intelligent selection instead of mass testing

The turning point came with a project in cooperation with the Technical University of Munich under the leadership of Professor Alexander Pretschner. By employing a PhD student, the team focused on developing a system that only selects relevant test cases based on the changes in the code. This approach not only allowed the team to save significant time, but also to use resources more efficiently and provide faster feedback to developers. Silke shared exciting details about the process of selection, including nightly analysis and automatic mapping of code changes to test cases.

From hours to minutes

The result of this project is impressive: in many cases, feedback to developers was reduced to 10 to 15 minutes - a huge improvement on the original seven hours. What is particularly interesting is the fact that this was achieved across different technologies. The increase in efficiency affects both Java and C++ areas of the company, demonstrating the broad applicability of the system developed.

What comes next?

During our conversation, Silke gave an outlook on possible future improvements to the system. These include, for example, prioritization within the selected tests in order to provide even faster feedback or a more detailed analysis at functional level for further efficiency gains. This continuous optimization clearly shows the potential and necessity for innovation in the field of software testing.

Conclusion

The story of IVU and its successful implementation of intelligent test case selection is a great example of technological progress. It impressively demonstrates how bold decisions and innovative collaboration can overcome challenges that many would consider insurmountable.