Test Intelligence

Historically grown test suites have a paradox: they test too much and too little at the same time - redundant end-to-end testing with untested code changes. Test Intelligence combines version control, coverage data and static analysis to select exactly the tests that are really relevant after a change. The result: finding 80% of defects in 1% of the time, making conscious decisions about test gaps and populating expensive test environments only with quality-assured software. Even teams with comprehensive code reviews are unable to cover all relevant changes in software testing without dedicated tools.

Podcast Episode: Test Intelligence

We take a trip into the world of test intelligence, coverage profiling and test gap analysis. How do you analyze development and test artifacts to efficiently identify errors? How do you evaluate the discrepancy between expected and actual results? And how do you incorporate all of this into the pipeline? Analysis tools promise continuous improvements in test quality and efficiency. But what is behind them?

“I don’t lose weight from weighing alone, unfortunately. And measuring alone doesn’t make me any better.” - Elmar Jürgens

Dr. Elmar Jürgens did his doctorate on static code analysis and was awarded the Ernst Denert Foundation’s Software Engineering Prize for his doctoral thesis. He is co-founder of CQSE GmbH and has been helping teams improve their quality assurance and test processes for ten years. Elmar regularly speaks at research and industry conferences and has received several awards for his presentations. In 2015, he was appointed Junior Fellow of the German Informatics Society.

Highlights der Episode

• Test gap analysis shows untested code changes before release by combining version control and coverage.
• Coverage profilers also record manual testing - the tool doesn’t care who is operating the software.
• Test impact analysis finds 80% of errors in 1% of the time through change-based test selection.
• Even code review in a small team of computer scientists cannot achieve complete test coverage without dedicated tools.
• Production coverage shows years of unused code that can be deleted instead of repaired.

From test gap analysis to test suite minimization: Test Intelligence

In this episode, I share my insights from an insightful conversation with Elmar Jürgens about Test Intelligence. We uncover the importance and application of analysis techniques in the development and testing process to find bugs more efficiently and improve the quality of software development.

Test Intelligence

Today I had the pleasure of welcoming Elmar Jürgens to the podcast. We talked about the topic of test intelligence. This term may seem abstract at first, but it encompasses a range of analysis techniques that aim to find bugs in less time and thereby significantly improve the software development process. From static analysis to coverage profiling, our conversation covered a wide range of topics that are all brought together under the umbrella of Test Intelligence.

The test gap analysis

One of the most prominent topics of our conversation was undoubtedly test gap analysis. This technology enables teams to uncover untested changes before the release. By merging data from the version control system and coverage profilers, a complete documentation of all test steps is created - a significant step towards improving quality assurance. It is particularly impressive that not only automated tests are taken into account, but also manual ones - a decisive extension of the conventional understanding of test coverage.

Coverage beyond automated tests

Elmar emphasized that coverage profilers have been used primarily for automated tests for years. The fact that we can now successfully use these technologies for manual testing marks a significant step forward. This development clearly shows that the boundaries between manual and automated testing are blurring and we are moving towards a more comprehensive view of testing.

First-hand experience: surprises from the field

Elmar’s experience with various projects revealed some surprising insights: In one case, particularly low test coverage even led to the termination of a project. Such moments highlight the importance of thoroughly analyzing and monitoring test coverage. It also shows that even in technically skilled teams, not all changes are always adequately tested - another argument for the use of test intelligence.

A look into the future

Our conversation also provided an exciting outlook on what we can expect in the future in the field of test intelligence. From using the same technologies in production to introducing new analytics such as test smells or requirement smells, the possibilities seem endless. This continuous progress promises not only more efficient error detection and correction, but also an overall increase in quality in the software development process.

Static and dynamic analyses combined

One particularly interesting aspect is the combination of static and dynamic analyses. By merging both approaches, we can get a more precise picture of where our code stands and what needs to be improved. These hybrid analyses represent an innovative approach to better support both developers and testers and ultimately develop better software.