Test estimation for a requirements-based system test only works if the test effort can be derived from the requirements. If you want to estimate with test points, those test points have to be traceable to the requirements documentation. If test cases are the basis for the estimate, they have to follow from the requirements. A text analyzer tool for English and German documents counts both, test points and test cases. The counts then go into a modified COCOMO-II formula that estimates the effort and duration of the planned test, taking into account factors such as the testability of the system, the degree of test automation and the test conditions. We recommend making at least three estimates and comparing the results. The outcome should be a cost range with a lower and an upper limit, which you then use to negotiate the scope of the test with the user. In the end, the scope of testing is a question of ROI.
Planning the System Test: Forward or Backward
Estimating test costs is one of the core tasks of test planning. There are two ways to approach it.
With forward planning, you estimate how much effort it takes to confirm that every requirement is met. From that you calculate the minimum duration of the test. Only then is the test budget aligned with the target costs and the end date with the calculated duration. This is how test managers would often like it to be.
In practice, backward planning is common. Here the budget and the deadline are fixed first. The estimate then answers how many test cases or test points can be tested with that budget by that date. Priorities follow from a risk analysis and a cost-benefit analysis. The goal is to test as much as possible within the budget and time limits.
Both approaches use the same estimation formula, just with different parameters. Which one applies depends on the situation.
Measuring the Test Scope from Requirements
Both approaches need a measure of the test scope. For a requirements-based system test, that means measuring the requirements. The requirements documents are analyzed, manually or automatically, to count the test cases or the test points.
Test cases are derived either from the requirements or from the use cases. Wherever the text requests an action, queries a state or sets a condition, a test case has to be specified. A condition even produces two test cases, one for the positive and one for the negative outcome. This lets you count every test case needed to cover the requirements text. Of course more test cases will come later, especially where the requirements are vague or incomplete. But you have a starting point. And if the count is automated, you get to a test case figure quickly and with little effort.
Test points work the same way. Like function points, they are counted from inputs, outputs, interfaces and database tables. If the requirements document mentions the user interfaces and system interfaces, they can be classified and weighted. The number of database tables and their attributes comes from the data model. Adding up the user interface, system interface and database points gives you the test points, and this count can be automated as well. That leaves us with two different measures of the test scope:
- one based on the internal logic of the system, and
- one based on the external system interfaces.
Measuring Test Productivity
Once we know how big the test project is, the next step is test productivity: how many test cases or test points can be handled per person-day. These are empirical values, and you cannot pull them out of thin air. The most valuable asset of a professional test organization is its memory, meaning the experience gathered in past test projects. That experience has to be protected and kept up to date.
Borrowing productivity figures from the literature or from other projects rarely works, because every environment has its own productivity. The reasons are partly cultural, partly technical and partly operational. In one project, for example, we measured an average of 3.8 test cases per day, in another 8.2. The first mostly tested web interfaces, the second batch processes.
In one of the author’s projects, 79,000 test cases were executed per release within two months by 92 testers. That comes to 21 test cases per tester-day. Keep in mind, though, that this was mostly a regression test. Only the changes and enhancements were tested against the specification. Everything else was tested against the previous version. Today the same test organization runs 120,000 test cases in six weeks with 24 testers. That is a productivity of no less than 166 test cases per tester-day. Automating the entire test process made this remarkable increase possible.
These numbers show how hard it is to define average values for test productivity. An automated test is far more productive than a manual one.
Measuring Testability
Not every application system is equally testable. Some are easy to test, others are hard. A system in which the application logic is tangled up with the user interface is harder to test than one in which the two are separate. The components of the second system can be tested through a batch interface, while those of the first can only be tested through the user interface. Systems with broad import and export interfaces are also harder to test, because testers have to generate more parameters with more possible combinations. Database size matters too. The more attributes a database has, the more data has to be generated and validated.
Testability can be measured through static analysis of the program source code, the user interface definitions (for example HTML or XSL sources), the system interface definitions (for example XML or IDL sources) and the database schemas in SQL or XML. Analyzing these software artifacts yields a rational measure on a scale from 0.1 to 0.9. Dividing the mean value of 0.5 by the measured value, say 0.45, gives the testability multiplier, in this case 1.11. In other words, the below-average testability of this software will raise the test effort by 11 percent.
Test Estimation with a Modified COCOMO-II Formula
Once test scope, test productivity and testability are known, calculating the test effort is simply a matter of applying the formula. We use a modified COCOMO formula here. The original COCOMO-II formula is:
The system types are standalone = 0.5, integrated = 1, distributed = 2, embedded = 4.
System size can be expressed in statements, function points, object points, use case points or any other unit.
Productivity is the number of size units a developer can produce per month.
The influence factor is the product of 20 individual influence factors on a scale from 0.7 to 1.4.
The scaling exponent is the mean of five project conditions:
- familiarity with the target architecture
- team cohesion
- quality of the development environment
- degree of reuse
- process maturity
It ranges from 0.91 to 1.23.
For estimating test projects, the authors propose expressing system size and productivity in test cases or test points, replacing the influence factor with the testability multiplier and using these scaling exponents instead:
- familiarity with the application
- team cohesion
- quality of the test environment
- degree of test automation
- test process maturity
The system type values are reduced to 0.5, 1, 1.5 and 2.
On this basis, a distributed system with 1,200 test cases and a testability of 0.45, tested by a team with a productivity of 8 test cases per tester-day and a scaling exponent of 1.05, would cost 320 tester-days.
Which Cost Drivers the Test Team Can Influence
The estimate shows what matters if you want to keep test costs down. Testability should be as high as possible, but the test team usually has little say in that. All the testers can do is make the user aware of it. The system type is a given as well.
What testers can influence are the test conditions and their productivity. They improve the conditions with professional testers who work well together in a mature test process with a high degree of automation. They raise productivity with more test automation. What counts in the end is running as many test cases as possible in as little time as possible while finding as many defects as possible. And that means a high level of test automation.
All of this points away from the traditional, homegrown way of testing software systems. Users need to be willing not only to have their application systems built by professionals, but also to have them tested by well-equipped professionals.
Frequently Asked Questions
What is the difference between forward planning and backward planning when estimating test costs?
Forward planning first estimates the effort required to validate all requirements, calculates the minimum duration based on that, and sets the budget and deadline accordingly. Backward planning sets the budget and deadline and determines how many test cases or test points can be accommodated within those parameters. Afterward, risk and benefit analyses determine the priorities. Both methods use the same estimation formula, just with different parameters.
How do you determine the number of test cases from a requirements document?
A test case must be specified wherever an action is required, a state is queried, or a condition is set. For conditions, there are two: one for the positive outcome and one for the negative outcome. This allows you to count all the test cases needed to cover the requirements text. Additional test cases are added later, especially when requirements are superficially or incompletely formulated. This count provides a starting point.
Can productivity metrics for testing be adopted from the literature or other projects?
No. Every environment has a different level of productivity, partly due to cultural factors, partly due to technical factors, and partly due to operational factors. In one project that was measured, an average of 3.8 test cases per day were processed; in another, 8.2. The difference stemmed from the type of system being tested: in one case, many web interfaces; in the other, batch processes. That is why the “memory” of a testing operation (the accumulated experience from its own projects) is so valuable.
To what extent can test automation change test productivity?
The difference can be very significant. In one testing organization described, 79,000 test cases were executed in two months with 92 testers per release, which equaled 21 test cases per tester-day. Later, the same organization managed to complete 120,000 test cases in six weeks with 24 testers, that is, 166 test cases per tester-day. This was due to the automation of the entire test process.
What makes an application system difficult to test, and how does that affect the effort required?
Systems in which application logic and the user interface are intertwined are difficult to test, because testing can then only be performed via the interface. Extensive import and export interfaces, as well as large databases with many attributes, also increase the effort required. Static analysis of source code, user interface, system interface, and database definitions yields a measure between 0.1 and 0.9. A value of 0.45 indicates an 11 percent increase in effort.
Can COCOMO-II be used to estimate test projects?
Only in a modified form. The 2010 article proposed expressing system size and productivity in terms of test cases or test points, replacing the influence factor with the testability multiplier, and adjusting the scaling exponents to account for test conditions: familiarity with the application, team cohesion, quality of the test environment, degree of automation, and test process maturity. The system type is reduced to values between 0.5 and 2.
Which factors affecting test costs can a test team actually influence?
Testability and system type are largely predetermined for the test team; at most, it can raise user awareness of them. The test team can influence the test conditions and productivity: well-coordinated, professional testers, a mature test process, and a high degree of automation. The goal remains to run as many test cases as possible in a short amount of time while uncovering as many defects as possible.
Why should the testing effort be estimated multiple times rather than just once?
It is recommended to perform at least three estimates and then compare the results. This yields a cost range with lower and upper limits. This range serves as a basis for negotiation with the user regarding the scope of the test. Ultimately, this scope is a question of return on investment, not a question of technically achievable completeness.
