Acceptance Testing

Acceptance testing is the last logical test level in the software development process and evaluates whether a system is good enough for use. Its aim is not to find errors, but to build trust: Quality should emerge beforehand, not still be tested in during acceptance testing. Those who get involved at an early stage save considerable costs in the end.

Key Takeaways

• The aim of acceptance testing is to create trust, not to find errors. Those who use it as the final quality assurance pay the highest bug-fixing costs in the entire development process.
• Testers who help shape requirements from the beginning increase the likelihood of building the right thing because they introduce testability, edge cases and non-functional testing criteria before the first line of code is written.
• Excessive acceptance testing is usually a symptom of a lack of transparency about what has already been tested in upstream stages, not a sign of thoroughness.
• Acceptance testing in an agile context and classic acceptance testing pursue the same goal, but differ in how deeply the test is woven into the development process and how early all those involved work together.

What is acceptance testing and what is it for?

From the perspective of the client, the specialist side or the users, acceptance testing checks whether a test object is good enough for use. Logically, it is the last test level after component, integration and system testing.

The central difference to system testing lies in the direction of view. System testing looks at the integrated system from the manufacturer’s or contractor’s perspective. The acceptance testing changes sides and takes the user’s perspective.

Unlike the upstream test levels, acceptance testing is not primarily aimed at finding errors. It is a confidence-building measure. At its core, the goal is to find no more errors because the test object does what it is supposed to do.

Florian Fieber, co-author of the book “Basiswissen Abnahmetest”, describes acceptance testing as the most exciting test level. It is particularly good at showing what you can do right or wrong in testing.

Acceptance testing in agile and traditional environments: same goal, different characteristics

In terms of the goal, acceptance testing and acceptance testing are the same. The difference lies in how well the activities are woven into the development process.

In an agile context, acceptance testing is built into the development process. Build-in-quality, shift left and cooperation between roles are more pronounced there. As a result, acceptance takes place earlier, more collaboratively and more efficiently.

In the classic waterfall or V-model project, acceptance testing follows the same principle, but struggles with different conditions. It often takes place very late. Requirements definition, development and testing are decoupled, and the test is added late.

The substantive idea remains the same. What changes is the degree of efficiency: the better the acceptance is interwoven into the process and collaboration, the more efficiently it can be played out.

Why timing determines success

The time at which testers are involved in the acceptance process determines almost all further possibilities. It determines how well acceptance testing can be dovetailed with development.

The bad case looks like this: Development has been going on for a long time, the requirements have been written for a long time, the implementation is being fine-tuned. Only a few weeks before delivery does the preparation for acceptance begin. This is exactly what you want to avoid.

This is a common error in thinking. Logically, acceptance testing is the last test level, but this does not mean that you have to deal with it last.

You don’t have to wait until the system testing is complete and the delivery has arrived to think about test planning, test scope and test cases. This work can be done much earlier.

If the tester is involved right from the start, he or she gets involved in the requirements process. Ideally, not as a guest who asks to have a say, but in an environment that provides for joint work on requirements. This requires organizational framework conditions that are easier to provide in an agile environment than in separate silos.

What makes a good requirement for acceptance

First of all, a good requirement is testable. The test view must be incorporated into the requirement at an early stage so that it can be tested at all.

Testers bring their own perspective to this. They ask questions, think around corners, consider edge cases and want to know exactly where the boundary between two cases is. This precision uncovers gaps before code is created.

A certain degree of completeness is part of testability. A user story needs not only the good cases, but also the edge cases and the negative cases.

The non-functional requirements are at least as important. Usability, accessibility, security and performance can be anchored early on using suitable acceptance criteria. This is where a tester can provide real added value for the team.

The ideal case: before the first line of code is created, business analysis, testing and development work out a common understanding. Acceptance criteria and test cases created in advance provide concrete examples and increase the likelihood that the right thing will be built.

The ideal acceptance testing is the one that doesn’t even have to take place

Acceptance testing that tests quality into the product comes at the worst possible time. That’s exactly what you don’t want.

In practice, acceptance testing is often much more extensive than necessary. The reason is rarely the test object itself, but a lack of trust and transparency. If you don’t know what has already been tested, you test everything again to be on the safe side.

This mistrust grows with decoupling. When a supplier or an external development department throws software over the fence, acceptance testing tends to get out of hand. In the end, quality is tested at the very end.

The ideal acceptance testing is one that doesn’t have to take place at all. The goal is to create trust, and I don’t create trust by finding errors. The errors should be found beforehand.

Florian Fieber

The argument can be made on the basis of the relative costs of correcting errors. Acceptance testing is a poor candidate for uncovering and fixing many bugs at this late stage. It can be effective as a last line of defense. But it is not efficient.

This does not mean that you should stop testing in acceptance testing. You keep testing, but what you test works. If the steps leading up to it run smoothly and are transparent, the acceptance test can be relaxed instead of taking the cow off the ice at the end.

How testing expertise and specialist knowledge come together in acceptance testing

Acceptance testing is often carried out by specialists who know the product from a user perspective but are not test experts. How much testing expertise is required depends on what the acceptance testing is supposed to achieve.

If the acceptance testing is to seriously uncover errors, it needs a combination of both views. Not every user is familiar with test techniques, different test characteristics and suitable test data. The test perspective and user perspective must work together here.

If, on the other hand, the focus is purely on confidence-building measures, the situation is different. Beta testing or a form of exploratory testing may suffice, where users use the test object in their normal work instead of working through formal test cases. In this case, strong testing expertise is not absolutely necessary.

Another decisive factor is whether the users see the test object for the first time during acceptance testing or whether they have been involved in the development process throughout. In the second case, it is only a matter of a final check.

Test-first as a lever: ATDD and BDD in acceptance testing

Test-first approaches shift acceptance to the left and sharpen the common understanding before implementation. Acceptance-Test-Driven Development (ATDD) and Behavior-Driven Development (BDD) are the central tools for this.

The effect: test cases that are created before implementation provide concrete examples. These examples facilitate understanding and support implementation because developers know more precisely what they are working with.

For this to be effective, the activities of business analysis and acceptance testing must be brought together. The roles of business analyst and tester remain different, but they belong together in terms of content and should work closely together.

Modeling of business processes and business rules as test support

Models serve two purposes in acceptance testing: they specify the test object and provide the basis for deriving expected behavior. BPMN is ideal for business process modeling.

Business rules can be mapped using DMN (Decision Modeling and Notation) and decision tables. From such models, you can derive which paths need to be covered in a critical business process.

The value of modeling lies not only in the specification. From a testing perspective, it helps to identify expected behavior and specifically test those paths that are critical from a business perspective.

The following overview organizes the focal points of the Certified Tester Acceptance Testing syllabus on which the book “Basiswissen Abnahmetest” is based: