Risk-based testing

Risk-based testing means concentrating testing efforts where an error causes the most damage. Three factors determine the risk: damage potential, detectability and complexity. If you multiply these factors, you get a key figure. The higher it is, the more test coverage is required in this area.

Key Takeaways

• At the Bison Group, risk-based testing has led to 50 to 60 percent of the old end-to-end testing being eliminated without replacement because it was shifted to lower test levels or recognized as superfluous.
• Risks are best broken down into domains, for example sales, incoming goods or master data, and then specifically name what must not go wrong under any circumstances.
• The evaluation of each risk from three factors, damage potential, detectability and complexity, each multiplied on a scale of 1 to 3, results in a key figure that makes prioritization decisions objectifiable.
• Risk-based testing creates quality awareness throughout the entire development chain, from requirements analysis to implementation, because everyone involved can see where errors cause the most damage.
• A risk catalog is not a one-time document, but must be re-evaluated with new requirements and after each bug feedback in order to retain its control effect.

Risk-based testing starts with a headline

The quickest introduction to risk-based testing is a simple question: Which headline do you not want to read in the newspaper tomorrow? From this headline you formulate a risk, and from the risk you derive where testing needs to take place.

The example behind this is tangible. An invoice run produces 4,500 invoices, and somewhere along the line the system adds up incorrectly or applies the wrong tax rate. The customer has to get the invoices back, and the damage is high on both sides. It is precisely cases like this that mark the point at which testing is worthwhile.

The advantage of this question: the customer knows best where it hurts him the most. If you ask them directly, you have often already determined your damage potential before any test case has been written.

How a risk is assessed

At the Bison Group, risk assessment rests on three components: damage potential, detectability and complexity. Each is scored on a scale of 1 to 3 and the scores are multiplied. The higher the resulting score, the closer you need to look.

Uwe Paesch describes the logic using the invoice example. The potential for damage is high because the customer has to retrieve thousands of invoices. The detectability is high because the error is not visible at first glance. Complexity is high because the error is difficult to find. Three high values result in a high risk.

A high overall value is not the only signal. If you score high everywhere, you automatically get high numbers. Therefore, keep an eye on the risks with a high potential for damage, even if they are quickly noticed and easily rectified. If a fault like this goes to the customer, they will still ask three times whether it is still possible.

How to cut risks at the right altitude

The first step is not the assessment, but the cut. Define topics first, then it’s easier to include the risks underneath.

With an ERP system, domains are ideal: Sales, ordering, goods receipt, delivery, interfaces, master data. In the case of a mobile app, the functionality of the app itself and communication with the peripheral systems. Only once this cluster has been established do you consider what must not go wrong in each area.

The most common pitfall is the wrong altitude. “The sale doesn’t work” is too coarse a risk, individual details are too fine. If you don’t hit this level, the assessments are useless, one too low, the other far too high. For an ERP system, you end up with around 150 risks, in the area of sales around a dozen.

If you are starting out, you should start small. First, specialty areas, then break it down. The first steps count more than the perfect catalog.

Risk-based testing is particularly worthwhile during a migration

A migration is the ideal occasion to take a risk-based approach. Bison was faced with exactly this case: replacing a rich client with a web client, plus around 1,000 end-to-end tests that needed to be transferred.

Instead of replicating every old test one-to-one, the decision was made to provide test coverage where it would hurt the customer the most. They drew up a risk catalog, identified the highest risks and involved business people, developers and architects in the assessment. Only then did they check which tests already existed, whether they needed to be expanded and where new test cases were needed.

The result was a noticeable reduction in workload. Depending on the system, between 50 and 60 percent of the old end-to-end testing was eliminated because the risk coverage showed that it was simply not needed.

Why end-to-end testing is not the standard answer

End-to-end testing is expensive to produce, expensive to maintain and expensive to provide feedback on. Before you write one, it’s worth asking: Do I really need it at this level?

If an end-to-end testing only tests one small thing at the end, it belongs two levels lower. If you set the test one or two levels lower, you will get your feedback much faster and save on maintenance costs.

With Bison, automation is already running tightly. All unit and module tests run with every build, followed twice a day by complete tests including end-to-end testing. This shifts the real work to the question of the right test level, not whether automation is needed.

Risk assessment is a living process, not a law

A risk catalog is never finished. The assessments change with the feedback that comes back from the field.

This feedback comes from several sources: from requirements engineers, from the customer and from bugs. At Bison, a violated risk was initially noted for each bug, initially in Confluence catalogs, later directly in Jira, so that the linking is sustainable. This revealed a correlation: high risk index, higher bug level.

New requirements also trigger a re-evaluation. If an idea comes from the customer, you check whether you are introducing a new risk or violating an existing one and go through the assessment again.

Over three years, this has provided a clear picture. Blocker and high bugs, i.e. the errors that hurt the customer, show a clear downward trend.

How to reach an agreement in the event of differing assessments

If one person sees a major risk and the other sees none, there is a discussion, not a vote. The arguments are heard and then a consensus is sought.

This usually happens quickly and objectively. In the end, someone says that they can live with the others’ assessment. Only a genuine veto forces a longer discussion. Friction arises above all in terms of detectability, i.e. how quickly an error becomes visible, and in terms of complexity, where scholars argue.

The biggest side effect is awareness of quality

Risk-based testing has an impact beyond the test team. It creates an awareness of where to look closely, and this runs through the entire chain.

It’s not just the developer who sees that three risks are assigned to a story and then tests particularly carefully. It goes right through the chain, right up to the requirements engineer who talks to the customer.

Uwe Paesch

In concrete terms, this means that the requirements engineer can tell the customer that a certain implementation will ruin the pricing. Developers understand better how complex a function really is for the customer instead of just working through their own story in their own bubble. The risk discussion turns into aha moments about what is happening outside.

Where risk-based testing goes next

It takes time to spread throughout the company. At Bison, the approach began in the ERP teams, which were driven by the legacy portfolio, then other teams were gradually added. It will take time for this to take effect across the board.

The path there did not run smoothly. in 2022, the goal was set to implement risk-based testing across the entire Group, which was too ambitious. in 2023, it was set and accompanied by a course that teaches the procedure, purpose and method.

The next milestone is the comprehensive introduction, followed by more automation. Not just tests that run automatically, but tests that are generated from use cases and risk descriptions or created as suggestions. This is exactly where AI support is expected in the coming years.