The acceptance test is the highest of the test levels. The theory says that here the customer tests and realizes with a smile on his face how wonderfully all his requirements have been implemented. Then he gives his ok, the contract is fulfilled and everyone is satisfied.
In practice, unfortunately, it usually looks a bit different. Crashes, unclear operation, missing or wrong functions. Instead of a satisfied smile, there are outbursts of anger, frustration and arguments about what was meant and how.
One way to prevent this is to use the previous test levels such as unit testing, integration testing and system testing sufficiently. Or involve the customer as early as possible. This is also a key driver for agile methods and agile testing. Early, short feedback loops and reviews by the customer and user. All of this helps to avoid a rude awakening in the end.
Definition acceptance testing
The ISTQB defines acceptance testing as: "A test level focused on determining whether a system can be accepted."
Whether this definition is helpful is for everyone to decide. In essence, the difference to previous test levels is that the customer or user now acts as the tester. This of course brings a new perspective on the test object and thus new questions, opinions and ideas.
For projects that do not run on a customer environment, e.g. a mobile app for the consumer market, alpha or beta tests are frequently used here.
Test base
The technical specifications of the customer, client or user serve as the test basis for the acceptance testing . Examples are business processes, user stories, epics, use cases, etc.
In the case of migrations, the operating instructions or documentation of the old system sometimes also serve as a test basis.
Since contractual acceptance is often also at stake here, the decision as to whether a test is successful or not has far-reaching consequences.
It is equally important to check non-functional requirements here and to confirm their implementation, e.g. by means of an operational acceptance testing.
Test case creation for acceptance test
Functional requirements are often available in text form. How can test cases be created from these requirements? Similar to system testing , it is best to combine structured test case methods with experience-based test case creation.
One problem here is that the customer or user does not know the structured test methods such as equivalence class formation, limit value analysis or decision tables. There is also often a lack of knowledge about how test cases are documented, structured and processed. It has proven useful to schedule a small training session at the beginning of the test activities to impart this basic knowledge. A few tips and ideas are enough to massively increase the quality of a department's tests.
Test object
The test object for the acceptance test is the integrated overall system. The application is tested from the outside, usually as the users will do it every day in the future.
Interfaces are already integrated and connected to the other systems. The goal is to test very close to the later production status.
Test objectives of the acceptance test
The goal of the acceptance test is to check whether the functional and non-functional requirements for the application have been met and sufficiently implemented. And this from the customer's point of view. The customer ultimately decides whether his ideas and concepts have been implemented as he imagined.
The result of the acceptance test is the final acceptance. This means, for example, that the project is finished and moves on to the next phase of maintenance. Payment flows, training and go-live also often depend on the outcome of the acceptance test.
Test environment for acceptance tests
A valid statement on the test result can only be achieved at acceptance testing if the tests are carried out in the customer's final infrastructure. This is the only way to exclude incompatibilities and side effects.
When a system is put into production for the first time, sometimes the future productive environment is also used for acceptance testing. In times of virtual systems, the provision and management of the infrastructure is much more time and resource efficient.
Test data
The test data should also be provided or at least controlled by the customer or client. Anonymized production data can be used. An alternative is to use the test data management of the system test. Regardless of which option is used, it is important to adopt the subsequent user perspective as well as possible.
Test automation of the acceptance test
Since the acceptance testing is usually only carried out once in classic project procedures, test automation tends to play a subordinate role here.
In agile acceptance testing, things look quite different. Here, test automation is mandatory today, as they have to be executed over and over again.
Robot-controlled process automation (RPA) is currently experiencing a trend in this context, which can be used from acceptance testing further into the production environment.
Acceptance testing in agile projects
The acceptance test is essential in the agile context. Here, the focus is very much on the customer's benefit, his feedback and his ideas. Due to the usually short iterations and the intensive exchange with the customer or later user, the acceptance test becomes an ongoing process. The constant feedback from the user is taken up and incorporated into the next steps. Of course, this also massively reduces the potential frustration factor at the end of the project. In some agile projects, each accepted partial step is considered to be completely finished and is only checked via automated regression tests. There is then no further manual testing for this.
Typical problems
In agile projects, I have hardly encountered any problems with acceptance testing. Quite the opposite. Everyone involved benefits from the intensive and ongoing exchange.
In classic projects with a acceptance testing at the end, the world unfortunately looks different. Here, challenges occur again and again:
- Non-functional requirements were not considered until acceptance testing and are now causing problems, e.g. usability testing, performance or reliability testing.
- The acceptance testing is completely based on the requirements and ideas of the customer or user. If gaps or a different understanding were not clarified in the system testing gaps or a different understanding were not clarified, these now come to light and have all the more consequences.
- Time pressure is often very high in the acceptance phase, as the project is supposed to be finished soon. This increases the stress for everyone involved and also their irritability.
- Business acceptance testers, end users or users sometimes lack the basics of software testing. How are test cases created? How are they documented? And how are they executed? In practice, this often leads to problems and delays.
acceptance testing in practice
I have been amazed to see projects where the acceptance testing consisted of three clicks. Then acceptance was explained and anything that didn't work was corrected via maintenance contracts. At the other end of the scale, I have seen customers who have set up professional test centers to perform acceptance tests. The range is wide, but so is the consistency.
Things can get heated and the air is thick in many an acceptance meeting. This wouldn't have to be the case if the months beforehand had been spent working together towards the moment of acceptance. It is therefore always a matter of concern to me personally to plan and carry out this phase together. Client and contractor.