Implementation as a test oracle

In the development of autonomous driving functions and safety-critical systems, coping with increasing complexity is a key challenge. A proven approach to quality assurance relies on so-called reference implementations: Testers create a stand-alone, simplified software version based on the requirements, independently of the developer code. In the subsequent back-to-back testing, the results are compared and deviations are systematically analyzed.

Podcast Episode: Implementation as a test oracle

In this episode, I talk to Stefanie Leitner about an exciting topic: How do you efficiently test highly complex software in the automotive industry? Stefanie works at VAIVA, a subsidiary of the Volkswagen Group, and reveals how she reduces testing effort with a clever idea: they build their own independent reference implementation for their tests. Sounds strange at first? But it works surprisingly well and saves a lot of time and nerves in the end.

"If you have to calculate the target values manually for 3000 signals, you have no chance in testing." - Stefanie Leitner

Stefanie Leitner has more than 13 years of experience in automotive testing. Since 2011 she has been working at VAIVA GmbH in various roles such as software tester, test manager or test lead for projects in the field of Advanced Driver Assistance Systems and Active Safety.

Stefanie Leitner is currently responsible for the strategic orientation of testing at VAIVA as Principal Software and System Testing. Her tasks include the implementation of standardized test processes and methods in accordance with ASPICE and ISO26262, as well as the development of new, innovative test methods.

Highlights of the Episode

• A reference implementation serves as an automated test oracle for the verification of complex software functions.
• Developers and testers work in parallel based on the same requirements, but independently of each other.
• Test cases compare outputs of the original software with the reference implementation in back-to-back testing.
• Reviews of the reference implementation ensure quality and confidence in the test oracle.
• Changes in requirements can be efficiently tracked in the reference implementation and tests.

Reference implementation as a test oracle

Introduction

Using a reference implementation as a test oracle is a crucial part of the software test process. This implementation serves as a comparison model for testers to check the correctness and quality of the developer code. By automating test cases using the reference implementation, testers can work more efficiently and detect errors more quickly.

An important aspect in this context is the minimum viable test strategy, which makes it possible to react quickly and effectively to changes in the test process. The development and use of reference implementations play an important role in ensuring the functionality and reliability of software products. In this article, we will dive deeper into the role of reference implementation as a test oracle and take a closer look at its benefits and applications.

The role of reference implementation as a test oracle

The use of a reference model plays a central role in the software test process, especially as a test oracle. A reference model is created by testers independently of the development team and represents the implementation of the requirements in accordance with the specifications. This model is used to automatically calculate and deliver the expected results for various test cases.

Automation of test cases through reference implementation

Automation of test cases is made considerably easier by reference implementation. Instead of manually determining target values, the reference model enables the automatic generation of target values for each individual test scenario. This not only saves time, but also increases the accuracy of error detection. Each input value runs through both the developer's actual software and the reference model. The output of both systems is systematically compared.

Comparison between developer code and reference model

The comparison between the developer code and the reference model takes place at signal level and includes all relevant output values. If the results match, the test case is considered passed. Deviations signal potential errors in the developer implementation, which can then be specifically investigated. The independence of the reference implementation is important here: testers do not have access to the source code of the developer software in order to avoid unintentional adoption of errors or misinterpretations.

Deep validation of complex algorithms

This approach allows deep validation of complex algorithms, such as those commonly used in systems for autonomous driving. Such systems require special test strategies and methods for software testing in the automotive industry to ensure that the software works according to the requirements.

Advantages of using a reference model

The use of a reference model as a test oracle brings significant advantages for quality assurance. The automated execution of tests based on the reference model ensures a comprehensive and consistent appraisal of the software, which largely eliminates manual sources of error. This leads to a significant improvement in quality assurance, as all relevant signals and states are continuously and precisely compared.

Efficiency in error detection and correction

The efficiency of error detection and correction increases significantly. As the reference model is created independently of the developer code, discrepancies can be clearly identified. Errors can be detected early on in the test process and addressed in a targeted manner. Feedback to the developers is faster and more substantiated, which facilitates the debugging process.

Automation of test procedures

A decisive aspect is the automation of test procedures. With the reference model, large quantities of test cases can be systematically executed without the need for manual adjustments for each variant. Changes to the requirements can also be efficiently incorporated into the reference model through targeted modifications, which significantly increases productivity in the test process. This structured approach also enables better traceability of requirements through to implementation and their test coverage.

Structured testing with defined test levels

A clear test strategy with defined test levels provides additional structure for tests and helps to test different areas of the software in a targeted manner.

• Automated tests ensure a high level of consistency
• Early detection of deviations between target and actual behavior
• Reduction of manual effort in test case maintenance
• Improved traceability through linking with requirements

These properties make the reference model an indispensable tool in demanding development environments, especially when complex algorithms and numerous signals need to be tested.

Implementation and application of the reference model

Various aspects play an important role in the implementation of a reference model in software development:

1. Implementation of the reference model in software development

Testers create a reference implementation based on the requirements and the architectural design. This makes it possible to emulate the functionalities of the developer code and use it as a test oracle.

2. Role of the reference model in code review and bug fixing

Back-to-back testing compares the reference implementation with the original code to ensure that the software works according to specifications. This helps with error detection and correction in the development process.

3. Ensuring traceability between requirements and implementation

Linking requirements to the original code and the reference implementation enables complete traceability. This helps to ensure that all requirements have been implemented correctly and are consistent.

The implementation and use of a reference model as a test oracle therefore plays a crucial role in quality assurance and error prevention throughout the entire software development process.

Future developments and enhancements in the area of software testing

With the increasing complexity of software systems, the integration of AI into the test process is becoming more and more relevant. AI support enables more efficient analysis of large amounts of data and helps to detect patterns and anomalies that human testers might miss.

The further development of the reference model is crucial in order to meet the constantly changing requirements. Through regular updates and adjustments, the reference model can be kept up to date to ensure an accurate and reliable test environment.

The future of software testing will be heavily influenced by technological advances. The integration of AI and the continuous development of the reference model are crucial to further improve quality assurance in software development and make it more efficient.

Conclusion: The future of quality assurance with reference implementations as a test oracle

Reference implementations as test oracles offer a precise and automated way of ensuring compliance between software and requirements. They enable in-depth testing and efficient error detection through the direct comparison of developer code and reference model. Challenges lie in the trustworthy creation of the reference models and in the required technical qualifications of the testers.

• AI technologies open up new potential for accelerating the development of reference implementations.
• Automation and intelligent analysis can further optimize test processes.
• Future developments will strengthen the integration of AI and thus sustainably improve quality assurance.

Frequently asked Questions