A powerful combination
After more than 40 years experience in software engineering still too many projects overdraw budget or fail altogether. Reasons for that lie, amongst others, in incomplete and changing requirements.
Although much can be done by having a dedicated requirements engineer in your team, this paper shows, how to improve even more by using the experience of software testers in detecting defects in requirements specifications in early project phases. This combination is very promising, as can been seen by current training opportunities for software engineers. The new successful super certification QAMP® (QAMP Quality Assurance Management Professional) comprises both, tester and requirements engineering skills amongst others.
Introduction
The Standish Group Chaos Report, recorded since 1994, is synonymous for visualizing the high rates of projects overdrawing time and budget or failing 1. Reasons for the low rate of successful projects have long been object of investigation and are sufficiently known to most successful project managers. While some problems have been reduced by use of engineering techniques and standards, the following still remain the most common 2.
- Unrealistic time and/or resource estimates
- Misunderstanding of the scope/objective/requirements
- Changing requirements during development
- Lack of client/end-user commitment/involvement
- Poor-Quality Work
- The belief in magic
As you see requirements are one major source of problems in this list. Not only by being misunderstandable and changing, but also by influencing other problem sources as time and resources estimates. As many engineering issues are considered solved using refined technologies from project structuring to testing methods the gap between customer’s and developer’s point of view has only in recent years become subject to increasing standardization. While notations and templates for requirements abound still for a long time there was no clear concept of how to elicit requirements in the first place. Who is to be interviewed? What is to be done, not to miss important stakeholders or influences? How do I detect presumed requirements? Methods for requirements elicitation combined with standards and best practice approaches are part the “Certified Professional for Requirements Engineering” (CPRE), a relatively new training for software engineers. With the structuring of this certification the International Requirements Engineering Board (IREB) follows closely the example set by the very successful ISTQB Certified Tester 34.
The Gain in Professional Requirements Engineering
At the beginning of each project there is the objective, however vague and diverse this objective may be in the minds of all people involved. In the requirements analysis phase project objectives are refined and finally set down as distinct requirements. The quality of these requirements depends very often on skill, personal objectives and support for the persons recording them. To reduce this influence, the IREB defines a dedicated role for requirements engineers. This leads to having a neutral person, that can identify and consider all stakeholders and their respective goals in a project. By careful use of his communication skills, templates and engineering methods the requirements engineer can elicit, document and prioritise all relevant requirements. Validation and verification of requirements is also a responsibility of the requirements engineer. Different documentation techniques as well as reviews can help to identify incomplete and defective requirements. Last but not least careful requirements management helps to keep track of changing requirements. The gain is obvious: an exhausting set of understandable and testable requirements details project objectives making it easier to plan, realize and test. It’s the CPRE methods we use, when first checking a new requirements specification for completeness and testability and not rarely we identify specification gaps and presumed requirements in that way.
The Real World
Given all the approaches and techniques in requirements engineering, no project should fail because of an incomplete requirements specification. Yet, in our experience, in many projects requirements are still specified by domain experts who are not familiar with refined specification techniques and often do not have the time and support to elicit all relevant stakeholders and details. While the fact that good and exhaustive requirements are a critical factor in project success may not yet be the focus of project management, the need for independent testing very often is. Some project failures generate a lot of media attention. The public conclusion is very often that systematic testing could have prevented problems. Therefore, a project team must not only consist of a professional requirements engineer, but of several test experts for systemand acceptance tests.
Learning curve - Step 1
In many of our projects we experience that customers still consider testing a phase following the implementation. So when we join the project and get an opportunity to analyse the requirements specification for the first time the system already is being developed. Being certified testers, we question requirements, that are not detailed enough for specifying test cases. Details the customer gives are usually unknown to the developers and sometimes even include new requirements grown out of deeper reflection on the system. While we may be happy with the received information developers are usually not. How could they, when the customer suddenly moves the goal. The followup are very often long discussions, overdrawn budget and time. Still the feedback for testers is usually great, as project lead appreciate our deep look into the requirements. How many of the defects in a project can be traced to faulty requirements illustrates the following chart.
Although requirements based testing can be very effective in detecting faulty requirements, resolving these defects is expensive compared to defects committed in later project phases.
Learning curve - Step 2
The obvious conclusion out of late testing is: defects in requirements must be detected earlier in project. One obvious solution: testing starts at the beginning of the project. This is quite often a solution realized by our customers after first experiences with requirements based testing. The fact, that this approach also is encouraged by the certified tester syllabus provided by the ISTQB, helps to push the matter for future projects. As a first step, testers are integrated in the reviewing process, before each requirements specification is released for design and implementation. Having experience in quality assurance makes testers very good reviewers for requirements specifications. Since it’s their job, to derive test cases from specifications, they first of all look into the testability of requirements. Are there criterias to tell whether a requirement is fulfilled or not? Are the requirements exhausting to describe the behaviour of the system? How much a review by testers can add can be illustrated by the number of questions raised: During a 9 month functional testing phase, starting while implementation was already under way, two testers raised about 250 questions, several of which led to an addition of functionality even to the extend of change requests, that is about one questing per tester per day. In another similarly complex project 2 testers reviewed a concept before the implementation phase started within 2 days. Being the first review they didn’t even start on deriving test cases yet. Still within those two days more than 60 questions were raised and could be solved by revising the concept thus saving many days of development and test effort. What is special about testers in reviews is the different point of view. While stakeholders very often just provide enough information to convey his intentions it is the testers objective to get clear and testable requirements for deriving test cases. If targeted behaviour in a test case can not be determined a tester will invariably demand details. E.g. while a customer may demand a sufficient performance, so the user can ‚work without interruptions‘ the tester will ask for precise respond times. Only with those it is possible to decide whether a test case is passed or failed. While testers by their experience are good in improving requirements a training in requirements engineering can improve skills in requirements elicitation and raise the awareness of presumed requirements.
Status Quo
The learning curve described is based on our experience in relatively small projects (each realized in less than two years time). Often the same customer will have different projects with different managers. Thus the learning curve includes common knowledge within a company. This is a typical status quo:
Testing, which once was the starting point of an initiative for quality in projects by now has become a fixed part of every project. The established testing process is based upon the recommendations of ISTQB and qualifications such as a test certification foundation level or advanced level are demanded for any new testing expert, thus establishing a common vocabulary and methodology. More than 100.000 certified testers in more than 40 countries prove, our experience conforms with global trends. Additionally we note an increasing focus on requirements quality. While a few years ago a domain expert would write down the requirements for a new system using a template grown out of his or his companies experience, the effort for requirements elicitation has increased dramatically. Early involvement of testers and reviews by different stakeholders are a step towards increasing professionalization.
Further improvement
As depicted above professionalization in requirements engineering is ongoing. Here we would like to describe, what we hope to achieve.
At least one dedicated requirements engineer per project
It is a truth universally acknowledged that independent testing yields a neutral and reliable image of software quality, by not siding either with development or with customer interests. The same truth should be applied for requirements engineering. Although still many customers demand domain knowledge, a dedicated requirements engineer with a neutral view can avoid defects introduced through domain experts that are blunted by habit. Additionally he can apply knowledge acquired in special trainings, such as CPRE. Common language in modelling requirements e.g. in UML simplifies communication with development and test.
Establishment of real requirements testing
System and acceptance tests are also referred to as specification-based or requirements-based tests. Many well-known test methods, such as use case tests or state transition tests, are requirements-based. What these methods have in common is that they are based on models. Although these methods are well known to certified testers, they are often not applied systematically.5
To achieve very good requirements testing, two approaches are necessary: First, the use of these model-based methods must be encouraged in the specification of test cases. Only this leads to reproducible test coverage. Secondly, the models must already be applied to the requirements in the review phase. Most model-based test methods are capable of detecting specification errors, a property that is often underestimated in the specification phase.
To give an example decision table testing can be named. Here the combination of conditions, relationships and constraints is tested. So when constructing test cases undefined results out of rare combinations can be detected.
Another example is state transition testing. While requirements quite often are documented in textual form, the derivation of a state model can be used to depict processes and behaviour in different states of a system. This way quite often additional requirements can be identified.
In the end application of already well known methods not only leads to the improvement of test coverage of requirements but also improves requirements quality.
Another benefit of common models in requirements engineering and testing is the dramatically improved tracability. Especially in safety critical systems it is necessary that requirements can be traced to their models and further on to the test cases verifying those models and back again. Using common models coverage metrics can give a detailed status, starting with the overall test coverage and leading to the coverage of each single model element and requirement. 6
Are we there yet?
Our goal in high quality requirements doesn’t seem to be to far. The awareness of requirements quality is increasing in customers and providers of software engineering. For a lasting effect a suitable training for all involved persons is essential. The training must not simply address single techniques, but should focus on connections between different project roles. Common modelling languages and an idea of objectives and work of other team members improve mutual esteem and enable the consideration of each others demands. The requisitions of cooperation should also be addressed in trainings. Already there are qualification schemes for a holistic approach to software quality, integrating requirements engineering and test. One of them is the new super certification QAMP® (Quality Assurance Management Professional) which is based upon CPRE and Certified Tester Foundation Level (CTFL®). Both trainings follow established approaches and are coherent. But QAMP even takes us further: although CPRE and CTFL are the groundwork of QAMP, evidence of experience and further qualification, such as project management, test management or secure software engineer are required. For periodic re-certifications up-to-date work experience, research activities or further qualifications are necessary. By that QAMP ensures the commitment to life-long learning of every certificate holder. 7
It is obvious that tools and methods suffice. Now it is our challenge to create awareness of the great benefit the combination of software test and requirements engineering can yield.
- The Standish Group International, Inc, http://www.standishgroup.com
- Watts S. Humphrey, Five reasons why software projects fail, Computerworld, 2002
- International Requirements Engineering Board (IREB) e.V., https://www.certified-re.de
- International Software Testing Qualifications Board A.I.S.B.L., https://www.istqb.org
- Sneed, Baumgartner, Seidl, The system testing, Munich, 2008
- McConnell, Upstream Decisions, Downstream Costs, Windows Tech Journal, 1997
- Quality Assurance Management Professional, https://www.asqf.de/asqf/produkte/qamp/