Requirements engineering and software testing belong together. Requirements testing starts when testers review new requirements for completeness and testability, and they find gaps before those gaps become expensive in design and code. This article from 2009 shows how projects learn this combination step by step.
Requirements Testing: Why Testers Should Review Requirements Early
After decades of experience in software engineering, too many projects still overrun or fail outright. Incomplete and changing requirements are among the reasons.
A dedicated requirements engineer on the team already achieves a lot, but this article goes one step further. Project quality improves even more when you use the experience of software testers to find errors in requirements specifications in the early phases of a project. This combination is very promising, as the training currently available to software engineers shows. The QAMP® certification (Quality Assurance Management Professional), for example, covers both testing and requirements engineering skills.
Why Software Projects So Often Fail on Requirements
The Standish Group’s Chaos Report, published since 1994, has become shorthand for the high share of projects that exceed their schedule and budget or fail. The reasons behind the low success rate have been researched for a long time and are well known to most successful project managers. Engineering techniques and standards have reduced some of the problems, but the following are still the most common:
- Unrealistic estimates of time and/or resources
- Misunderstood scope, goals or requirements
- Requirements that change during development
- Lack of commitment from the customer or end user
- Poor quality of work
- Belief in magic
As you can see, requirements are one of the main sources of trouble on this list. Not only because they are hard to understand and keep changing, but also because they affect other sources of trouble, such as time and resource estimates.
Many engineering problems, from project structuring to test methods, are considered solved thanks to mature techniques. The gap between the customer’s view and the developer’s view, however, has only been bridged by standards for a few years. There have long been notations and templates for requirements, but for a long time there was no clear concept of how to elicit requirements in the first place. Who should be interviewed? What needs to happen so that no important stakeholders or influences are overlooked? How do I uncover implicit requirements?
Elicitation methods combined with standards and best practices are part of the Certified Professional for Requirements Engineering (CPRE), a certification program for software engineers. In the way it is structured, the International Requirements Engineering Board (IREB) closely follows the model of the very successful ISTQB Certified Tester.
What Professional Requirements Engineering Adds
Every project starts with a goal, however vague that goal may be and however differently the people involved picture it. During requirements analysis, the project goals are refined and finally written down as concrete requirements. The quality of those requirements very often depends on the skills, personal goals and support of the people who record them.
To reduce that influence, the IREB defines a dedicated role: the requirements engineer. This gives the project a neutral person who can identify all stakeholders and their goals and take them into account. By using communication skills, templates and engineering methods with care, the requirements engineer can elicit, document and prioritize all relevant requirements.
Validating and verifying requirements is the requirements engineer’s responsibility as well. Various documentation techniques and reviews help to find incomplete and incorrect requirements. And careful requirements management keeps track of requirements as they change.
The benefit is obvious. A complete set of understandable, testable requirements spells out the project goals and makes planning, implementation and testing easier. We use exactly these CPRE methods when we first check a new requirements specification for completeness and testability, and this is how we often find gaps in the specification and implicit requirements.
How Testers Check Whether Requirements Are Testable
With all the approaches and techniques requirements engineering offers, no project should fail because of an incomplete requirements specification. Yet in our experience, many projects still have their requirements written by domain experts. They are not familiar with advanced specification techniques and often lack the time and support to identify all relevant stakeholders and details.
Project management may not yet see good, thorough requirements as a critical success factor. The need for independent testing, on the other hand, is very often on its radar. Some project failures attract a lot of media attention, and the public conclusion is very often that systematic testing could have prevented the problems. A project team therefore needs more than a professional requirements engineer. It also needs several test experts for system and acceptance testing.
Learning Curve: Step 1
In many of our projects, customers still see testing as a phase that comes after implementation. So by the time we join the project and get our first chance to analyze the requirements specification, the system is already being developed.
As certified testers, we then question requirements that are not detailed enough to specify test cases. The details the customer provides are usually unknown to the developers and sometimes even include new requirements that came out of thinking about the system more deeply. We may be happy with the information we get, but the developers usually aren’t. How could they be, when the customer suddenly moves the goalposts? The result is often long discussions and blown budgets and schedules.
Even so, testers usually get very good feedback, because project management values how deeply we look into the requirements.
Requirements-based testing can be very effective at finding faulty requirements. Fixing those errors at this point, however, is expensive compared with errors that appear in later project phases.
Learning Curve: Step 2
The obvious conclusion from testing late is that errors in the requirements have to be found earlier in the project. The natural solution: testing begins at the start of the project. Our customers very often take this step after their first experience with requirements-based testing. The fact that the ISTQB syllabus for certified testers also promotes this approach helps push the topic in future projects.
As a first step, testers join the review process before a requirements specification is released for design and implementation. Their experience in quality assurance makes testers very good reviewers of requirements specifications. Since their job is to derive test cases from specifications, the first thing they check is whether requirements are testable. Are there criteria that show whether a requirement is met or not? Do the requirements describe the system’s behavior completely?
The number of questions raised shows how much a review by testers can contribute. During a nine-month functional test phase that started while implementation was still under way, two testers raised about 250 questions, some of which led to extended functionality and even change requests. That is roughly one question per tester per day. In another project of similar complexity, two testers reviewed a concept within two days before implementation started. As it was the first review, they had not even begun to derive test cases. Still, they raised more than 60 questions in those two days, which were resolved by revising the concept and saved many days of development and testing effort.
What makes testers special in reviews is their different point of view. Stakeholders often provide just enough information to convey their intentions. The tester’s goal is clear, testable requirements from which test cases can be derived. If the intended behavior for a test case can’t be determined, a tester will inevitably ask for details. For example, a customer may ask for performance good enough that users can “work without interruptions”. The tester will ask for exact response times, because only with those can you decide whether a test case passes or fails.
Testers are good at improving requirements thanks to their experience. Training in requirements engineering can add elicitation skills and sharpen their awareness of implicit requirements.
Status Quo
The learning curve described here is based on our experience in fairly small projects, each completed in under two years. Often one and the same customer runs several projects with different managers, so the learning curve reflects what a company knows as a whole. A typical status quo looks like this:
Testing, once the starting point of an initiative for quality in projects, has become an integral part of every project. The established test process is based on ISTQB recommendations, and every new test expert is expected to hold a qualification such as Foundation Level or Advanced Level certification, so that everyone shares a common vocabulary and method. More than 100,000 certified testers in more than 40 countries (as of 2009) show that our experience matches the global trend.
On top of that, we see a growing focus on requirements quality. Only a few years ago, a domain expert would write down the requirements for a new system using a template that had grown out of their own experience or their company’s. Since then, the effort put into requirements elicitation has increased dramatically. Involving testers early and having reviews by various stakeholders are steps toward more professionalism.
Further Improvement
As described above, requirements engineering is becoming more professional. Here we describe what we hope to achieve.
At Least One Dedicated Requirements Engineer per Project
It is widely accepted that independent testing gives a neutral, reliable picture of software quality, because it sides neither with development nor with the customer’s interests. The same should hold for requirements engineering. Many customers still insist on domain knowledge, but a dedicated requirements engineer with a neutral view can avoid mistakes introduced by domain experts whom habit has made blind to them. In addition, they can apply what they learned in specialized training such as CPRE. A common language for modeling requirements, for example UML, makes communication with development and testing easier.
Establishing Real Requirements Testing
System and acceptance tests are also called specification-based or requirements-based tests. Many well-known test techniques, such as use case testing or state transition testing, are requirements-based. What these techniques have in common is that they rely on models. Certified testers know them well, yet they are often not applied systematically.
Very good requirements testing takes two things. First, testers need to be encouraged to use these model-based techniques when specifying test cases. That is the only way to get reproducible test coverage. Second, the models have to be applied to the requirements as early as the review phase. Most model-based test techniques can uncover specification errors, a strength that is often underestimated during the specification phase.
Decision table testing is one example. It tests combinations of conditions, relationships and constraints. While the test cases are being built, undefined results from rare combinations come to light.
State transition testing is another. Requirements are often documented as text, but deriving a state model lets you map processes and behavior across the different states of a system. This often reveals additional requirements.
In the end, applying techniques that are already well known improves not only the test coverage of requirements but also the quality of the requirements themselves.
Shared models in requirements engineering and testing bring another advantage: much better traceability. In safety-critical systems in particular, it must be possible to trace requirements to their models, on to the test cases that verify those models, and back again. With shared models, the coverage metric can report a detailed status, from overall test coverage down to the coverage of each individual model element and requirement.
Training for Requirements Engineering and Testing
The goal of high requirements quality no longer seems far off. Customers and software engineering providers alike are becoming more aware of requirements quality. For a lasting effect, everyone involved needs suitable training. That training should not only cover individual techniques but focus on how the different project roles relate to each other. Shared modeling languages and an idea of what the other team members do and aim for increase mutual appreciation and make it easier to consider each other’s needs. The demands of working together should be part of the training as well.
Qualifications for an integrated approach to software quality, combining requirements engineering and testing, already exist. One of them is the QAMP® certification (Quality Assurance Management Professional), which builds on CPRE and the Certified Tester Foundation Level (CTFL®). Both are based on established approaches and are consistent in themselves. QAMP goes further, though: CPRE and CTFL form its basis, but it also requires proof of experience and further qualifications, such as project management, test management or secure software engineering. Periodic recertification requires current professional experience, research activities or additional qualifications. This way QAMP makes sure that every certificate holder commits to lifelong learning.
The tools and methods are clearly there. Our challenge now is to build awareness of the great benefit that combining software testing and requirements engineering can bring.
Frequently Asked Questions
Why are requirements considered one of the main causes of failed software projects?
Poor requirements have a double negative impact: they are misunderstood, they change during development, and they also skew time and resource estimates. On the list of the most common sources of problems, they rank alongside unrealistic estimates, a lack of commitment from customers and end users, poor work quality, and a belief in magic.
Does a project need a neutral requirements engineer if domain experts are writing the requirements anyway?
Yes, because domain experts are desensitized by habit and overlook errors that a neutral eye would spot. In 2009, the IREB defined a specific role for this purpose: a person who identifies all stakeholders and their goals, and who gathers, documents, prioritizes, and validates requirements. Furthermore, subject matter experts often lack the time and support needed for a comprehensive requirements gathering process.
What happens if testers don’t review a requirements specification until during implementation?
Then details, and sometimes new requirements, come to light while the system is already being built. From the developers’ perspective, the customer shifts the goal, leading to lengthy discussions as well as budget overruns and schedule delays. Although requirements-based testing effectively uncovers faulty requirements, fixing them at this stage is costly.
How effective is a specification review by testers in practice?
In one project described, two testers reviewed a concept over two days before implementation began and raised more than 60 questions, which were resolved through a revision. This saved many days of development and testing effort. During a nine-month functional testing phase, two testers raised approximately 250 questions, roughly one per tester per day.
How can you tell if a requirement is not testable?
There is no criterion by which to determine whether the requirement has been met or not. For example, a customer might request a level of performance that allows them to “work without interruptions.” The tester, however, asks for precise response times instead, because only then can a test case be deemed passed or failed.
Can test design methods such as decision tables be useful even before implementation?
Yes. Most model-based test techniques uncover specification errors, a capability that is often underestimated during the specification phase. In decision table testing, undefined results from rare combinations become apparent during the construction of test cases. When deriving a state model from textual requirements, additional requirements regarding processes and behavior often become apparent.
How do shared models from requirements engineering and testing contribute to traceability?
The requirement, model, and test case are then linked to the same representation, allowing the path to be traced in both directions. This is particularly necessary in safety-critical systems. The coverage metric thus provides a detailed status, ranging from overall test coverage down to the coverage of each individual model element and each individual requirement.
Is it sufficient to train testers and requirements engineers separately?
No. Training should not only teach individual techniques but also the interrelationships between project roles and the requirements for collaboration. Shared modeling languages and an understanding of each other’s goals increase mutual appreciation. As an integrated approach, the 2009 article cited the QAMP certification, which built upon the CPRE and Certified Tester Foundation Level certifications and required proof of experience.
