Analytical Quality Assurance: Testing and Measuring Artifacts

Checking and Measuring Software Artifacts

Analytical quality assurance is a low-cost, resource-friendly way to check software artifacts such as requirements, UML diagrams, source code and test cases against predefined rules and to measure them for criteria like complexity, quality and quantity. Because it works on the artifacts themselves, it finds problems early, while they are still cheap to fix.

Five Approaches to Software Quality Assurance

There are five ways to go about assuring the quality of software:

• psychological
• constructive
• analytical
• empirical
• retrospective

The psychological approach covers everything that influences people to do better work: rewards, penalties, promotions and appeals to professional ethics. The constructive approach covers everything that improves working conditions, such as better tools, languages, techniques and processes, and workplaces that are as modern as possible (premises, communication infrastructure, tools, social networks and the like). The analytical approach covers all measures for controlling the artifacts that software engineers produce: checking and measuring documents, models, source code and test cases in order to spot quality deficiencies and deviations from the target and point them out to the authors. The empirical approach covers all testing activities, such as unit testing, integration testing, system testing and acceptance testing. Here the behavior of the finished software is observed in a test environment under controlled conditions to detect deviations from the expected behavior. The retrospective approach covers everything done to improve the quality of software artifacts after they have been completed: re-engineering, refactoring and reimplementation. Unlike developers removing individual deficiencies and defects during development, these are targeted actions on an entire system or subsystem, aimed at raising the quality of that system. The assumption is that the degree of improvement can be measured. This article looks at the third approach, the analytical one. The other four are just as important and should be pursued alongside it, but each of them would fill a book of its own.

Deficiencies vs. Defects: What’s the Difference?

Before looking at the analytical measures, we need to separate two terms: deficiencies and defects. A deficiency is the violation of a quality rule, a departure from the standard. The rule might be that code blocks must not be nested more than three levels deep, or that every input parameter should be checked for plausibility.

A deficiency does not necessarily affect how a software product behaves. Some deficiencies can certainly lead to a failure, for example when exception handling or security checks are missing from the code, or when use cases are not fully specified. Most kinds of deficiency, though, only get in the way of maintaining and evolving the software, for example when code modules are too tightly coupled or data names are chosen arbitrarily. Every software artifact is subject to the laws of software evolution. To keep their value and stay useful, artifacts have to be developed further all the time, and that costs money. If you want to keep those costs down, you have to make sure the software is, and stays, easy to evolve. That gives analytical quality assurance its two goals:

1. Prevent behavioral failures
2. Reduce the cost of further development

Defects, unlike deficiencies, show up in the software’s behavior: it does something other than expected. Either it crashes, or it produces wrong results. You can only detect this kind of misbehavior by running the software. That is why testing is called the empirical approach to quality assurance: the software is executed so its behavior can be examined empirically. It always comes down to a comparison. Actual behavior is compared with expected behavior, and any deviation is first treated as a failure.

Failures are caused by flaws in the code or the design, the defects. But the specification or expectation can also be wrong. In that case, it is not the code that is at fault but the specification. Either way, there is a gap between target and actual, and it has to be closed. The purpose of empirical quality assurance is to uncover such gaps, including in the performance and resilience of the software. Testing is indispensable, but it is a subject of its own and is not covered further here.

Analytical Quality Assurance Means Checking and Measuring

In theory, analytical quality assurance is a wide, almost limitless field. In practice, it can be boiled down to a few automated checks. Software usually consists of four kinds of objects on four semantic levels:

• Requirements documents at the specification level
• Design models (UML/ER) at the design level
• Source code at the code level
• Test cases at the test level

Requirements documents and source code are both text. UML models are backed by structured data in XML documents that record the design elements and their relationships. The UML diagrams are rendered from the XML and vice versa. Test cases are turned into text as well, namely test scripts that are interpreted or compiled to generate test data and validate test results. At first, though, test cases are usually stored in tables whose rows and columns hold the test case attributes.

Analytical quality assurance is about checking and measuring these four kinds of text. The various checking and measuring activities are grouped here under the umbrella term software audit. They used to be done by hand under names like design reviews, code inspections and test audits. At the end of the 1970s, the author Sneed was himself responsible for inspecting Nassi-Shneiderman diagrams and CHILL programs in Siemens’ EWSD digital switching system project. It took him about half a day to check a single module. The code inspection alone stretched over more than a year. In the age of agile development, nobody can afford that anymore. Analytical quality assurance has to be automated, just like empirical quality assurance, that is, testing. The objects to be checked and measured are pulled from the configuration management system and fed straight into the right tool.

Checking and Measuring Requirements Documents

Requirements documents are checked and measured with a tool for automated natural language analysis. The texts first have to be marked up, either with XML tags or with keywords. Keywords are the better choice, because they keep the document easy to write for the end user: it pays to build these words into the text from the start, where they also serve as signposts. Examples of such keywords:

These keywords (keywords in context) make the text elements identifiable. Besides the individual requirements, they mark business rules, business objects, actors, interfaces, user interfaces and use cases. In a semi-formal requirements text, the elements that apply across projects, such as business objects and business rules, should be documented first. Then come the project-specific requirements, functional and non-functional. After that come the system actors, system interfaces and user interface patterns. The use cases come last, as the bridge to system design. That way, the use cases can refer to the requirements they fulfill, the business rules they implement, the objects they process, the interfaces they serve and the actors that trigger them.

Case Study: A Calendar Function

The following example shows a simple requirements specification for a calendar function.

Functional requirements

• FUNC-REQ-01 (determine the weekday): The user enters an eight-digit date with year, month and day and gets back the day of the week.
• FUNC-REQ-02 (multiple languages): The day of the week can be returned in German, French or Italian.
• FUNC-REQ-03 (alignment): The text of the weekday can be left- or right-aligned.
• FUNC-REQ-04 (error handling): If the date cannot be converted into a day of the week, question marks ”??????” should appear in the response field.

Non-functional requirements

• NF-REQ-01 (response time): The response time for customer queries should be <= 1 second and for customer orders <= 3 seconds.
• NF-REQ-02 (load capacity): The service must be able to process at least 500 orders per hour without loss of performance.
• NF-REQ-03 (availability): The service must be available at least 95 percent of the time, 24 hours a day, seven days a week.

Business objects

• GO-01: Calendar
• GO-02: Appointments
• GO-03: Weekdays

Business rules

• GR-01 (handling implausible dates): If a date is not plausible, the weekday is filled with ”?”.
• GR-02 (calendar start date): The calendar assumes that January 1, 1901 was a Tuesday.
• GR-03 (language codes): The following language codes apply in the company: German = “1”, French = “2”, Italian = “3”, English = “4”.

Use case The text analysis tool checks whether all mandatory properties and references are present and consistent, for example:

• Use case does not fulfill any requirement.
• Use case refers to an undefined requirement.
• Requirement is not fulfilled by any use case.
• Business rule is not implemented.
• Business object is not referenced.

The text analyzer also checks whether every requirement sentence is worded correctly. Chris Rupp and her colleagues at SOPHIST GmbH have defined rules for how requirement sentences should be written, known as the SOPHIST rule set. According to these rules, every requirement should, for example, be uniquely identifiable, traceable and interpretable. Every requirement sentence should have a subject, an object and a predicate and avoid nominalizations (packing several actions into a verb turned into a noun). Typical rule violations include:

• Actions are phrased in the passive voice instead of the active.
• A state refers to several objects instead of just one.
• An abstract state is not clearly defined.
• A statement relies on an implicit assumption.

The SoftAudit tool can check compliance with rules like these sentence by sentence. Finally, the analyzer can count the text elements and build a requirements metric. It counts, among other things, the requirements, business rules, object references, use cases and the processing steps within the use cases. It also counts function points, data points and use case points. The goal is to measure the size, complexity and quality of the requirements documents.

Checking and Measuring the UML Model

To check and measure the design model, SoftAudit uses an XML parser that parses the XML schema behind the UML diagrams. While parsing, it recognizes and counts model types such as classes, objects, methods, attributes, parameters, activities and use cases. At the same time, it recognizes and evaluates relationship types such as association, inheritance, usage and inclusion. Model complexity is the ratio of model relationships to model elements. The more relationships there are relative to the number of elements, the more complex the model:

Model complexity = 1 - (model elements / model relationships)

Model size follows from the number of model elements. Model quality is the ratio of actual to target. For example, every class should have only a limited number of dependencies on other classes (degree of coupling) and a limited number of attributes relative to its number of methods (degree of cohesion). Every class should have at least one attribute and two methods. Sequence diagrams may only reference classes that are already defined in a class diagram. Activity diagrams may only contain steps of a specified use case, which in turn are defined as methods in a class diagram. This is how the consistency of the model is checked.

This is also where the first cross-check across semantic levels happens. You can check whether every business object in the requirements documentation also appears as an object in the object model, and whether every use case specified in the requirements document shows up in the use case diagrams. In other words, you can verify that the requirements document and the design model are consistent. Typical deficiencies in the design model are:

• The class is referenced but not defined (“Class referenced is not defined”).
• The entity name does not follow the naming convention (“Entity Name does not conform to naming rules”).
• The maximum number of parameters allowed for a method has been exceeded (“Maximum Number of Parameters to a Method is exceeded”).

The analysis of the UML model again produces two reports: a deficiency report listing the gaps between target and actual, and a measurement report with the design metrics (size, complexity and quality).

+-----------------------------------------------------+
|  R E Q U I R E M E N T      S I Z E      M E T R I C S  |
+-----------------------------------------------------+
| Number of Function-Points                  ======>  188 |
| Number of Data-Points                      ======>  961 |
| Number of Object-Points                    ======>  726 |
| Number of Use Case Points                  ======>  108 |
| Number of Test Case Points                 ======>  255 |
+-----------------------------------------------------+
+-----------------------------------------------------+
|  R E Q U I R E M E N T    C O M P L E X I T Y     M E T R I C S  |
+-----------------------------------------------------+
| Data Density                           ======>  0.236 |
| Functional Density                     ======>  0.100 |
| State Density                          ======>  0.386 |
| Conditional Density                    ======>  0.402 |
| Referential Density                    ======>  0.768 |
| Test Case Density                      ======>  0.440 |
| Overall Requirement Complexity Rating  ======>  0.388 |
+-----------------------------------------------------+
+-----------------------------------------------------+
|  R E Q U I R E M E N T    Q U A L I T Y    M E T R I C S   |
+-----------------------------------------------------+
| Degree of Completeness                  ======>  0.966 |
| Degree of Consistency                   ======>  0.874 |
| Degree of Stability                     ======>  0.896 |
| Degree of Changeability                 ======>  0.678 |
| Degree of Testability                   ======>  0.364 |
| Degree of Conformity                    ======>  0.941 |
| Overall Requirement Quality Rating      ======>  0.786 |
+-----------------------------------------------------+

Checking and Measuring the Code

Source code is the artifact that gets checked and measured most. There are already plenty of tools for this, such as “Software Sonar”, “FxCop” (.NET) and “PMD” (Java), which check no fewer than 180 different coding rules. The “SoftAudit” tool also checks source code in twelve languages, from IBM Assembler to Java and PHP. Some coding rules depend on the programming language, for example:

• Use the Equal Operator to compare objects.
• Classes should be declared final.
• Methods may only be called through a defined interface.

Other rules apply to a whole class of languages, object-oriented languages for example:

• Class attributes should never be declared public.
• Avoid data casting.
• Methods should not be nested more than four levels deep.
• Avoid nested classes.
• Every call to a foreign method should be wrapped in a try-catch block.

And some rules apply to every language, such as:

• Conditions should not have more than three clauses.
• Text literals must not appear in procedural code.
• Avoid multidimensional arrays.

Rules like these serve three purposes:

1. To make the code uniform.
2. To make the code easier to change.
3. To make the code more secure.

Experienced developers will always challenge individual rules, and that is a good thing. It prompts everyone to think about the existing rules and ask whether they are still current and still make sense. In the end, though, a development team has to agree on which rules apply to it.

+--------------------------------------------------------+
|  D E S I G N    C O M P L E X I T Y    M E T R I C S     |
+--------------------------------------------------------+
| Class Interaction Complexity             ======>  0.953 |
| Class Hierarchical Complexity            ======>  0.166 |
| Class Data Complexity                    ======>  0.633 |
| Class Functional Complexity              ======>  0.500 |
| State Complexity                         ======>  0.692 |
| State Transition Complexity              ======>  0.420 |
| Activity Complexity                      ======>  0.580 |
| Usecase Complexity                       ======>  0.562 |
| Actor Interaction Complexity             ======>  0.238 |
| Overall Design Complexity                ======>  0.500 |
| Average Design Complexity                ======>  0.524 |
+--------------------------------------------------------+
+--------------------------------------------------------+
|  D E S I G N    Q U A L I T Y    M E T R I C S           |
+--------------------------------------------------------+
| Class Coupling                         ======>  0.443 |
| Class Cohesion                         ======>  0.323 |
| Design Modularity                      ======>  0.705 |
| Design Portability                     ======>  0.347 |
| Design Reusability                     ======>  0.161 |
| Design Testability                     ======>  0.534 |
| Design Conformance                     ======>  0.773 |
| Design Completeness                    ======>  0.250 |
| Design Consistency                     ======>  0.563 |
| Design Compliance                      ======>  0.420 |
| Average Design Quality                 ======>  0.451 |
+--------------------------------------------------------+

Beyond these checks on individual statements, there are also rules for code modules as a whole. They limit the size of code modules, the number of encapsulated data attributes and the number of external relationships. These rules promote modularity and reuse. Other rules of this kind limit the number of parameters in an interface and the width of database views, which promotes testability. Rules for commenting, for indenting nested lines of code and for limiting line length make the code easier to understand.

So coding rules are not there to boss developers around. They make it easier for developers to work together, just as traffic rules keep road traffic in order. Checking the code statically and uncovering rule violations prevents a lot of potential problems. Above all, it keeps technical debt from growing beyond all measure. In his book “Clean Code”, Robert C. Martin stresses how important static code analyzers are for keeping code quality up.

Measuring the code goes hand in hand with checking it. The tool counts statements, statement types, data, data references, objects and code blocks. Metrics are then calculated from these counts:

• Size metrics such as function points, data points and test points.
• Complexity metrics such as cyclomatic, linguistic and referential complexity.
• Quality metrics such as cohesion and coupling, reusability, security, testability and maintainability.

On a rational scale from 0 to 1, code quality should of course be as high as possible and complexity as low as possible. These values are important reference points for assessing the software as a whole and give direction for improvement work. If you want to improve code quality through refactoring, you have to compare the quality of the code before and after. That requires quality to be quantified.

+--------------------------------------------------------+
| Quality metric               | before remediation | after remediation |
+--------------------------------------------------------+
| Degree of Modularity           ======>  0.265  |  ======>  0.501  |
| Degree of Portability          ======>  0.862  |  ======>  0.842  |
| Degree of Reusability          ======>  0.960  |  ======>  0.960  |
| Degree of Testability          ======>  0.304  |  ======>  0.548  |
| Degree of Convertibility       ======>  0.120  |  ======>  0.242  |
| Degree of Flexibility          ======>  0.880  |  ======>  0.880  |
| Degree of Conformity           ======>  0.148  |  ======>  0.563  |
| Degree of Maintainability      ======>  0.426  |  ======>  0.494  |
| Weighted Average Quality       ======>  0.495  |  ======>  0.628  |
+--------------------------------------------------------+

Checking and Measuring Test Cases

For a long time, testware was neglected, even though every software system depends on its test cases. Nobody thought of checking them, let alone measuring them. Testware was never seen as part of the actual software product and is delivered to customers only in exceptional cases. But the same argument applies to the requirements specification and the design model, and by now it is accepted that their quality has to be assured too. The same goes for testware.

Testware means the test procedures, test cases and test scripts used to test the software system and its components. Test scripts are like program code. They have a formal syntax and can be parsed like code, and there are rules for how they should be written. The problem, as with requirements, is that there is no binding language standard. Every test tool vendor offers its own scripting language. Still, anyone who wants to can write a tool to check and measure test scripts, and for some kinds of script, such as web service test scripts, ready-made analysis tools exist.

if (operation = "getWeekDay");
    if (response = "getWeekDay1Response");
        assert out.$ResponseTime < "1100";
        
        if (object = "return");
            assert out.P1_TT = "16":
            assert out.P1_MM = "10";
            assert out.P1_CE = "18";
            assert out.P1_JJ = {81:89};
            assert out.LANG_CODE = {1:3};
            assert out.DIRECTION = "L";
            assert out.DAY_NAME = "Mittwoch";
            assert out.RETURN_CODE = {1:3};
        endObject;
    endResponse;
endOperation;

SoftAudit assumes that test cases are stored in an Excel spreadsheet or a relational database table in which each column holds one attribute of the test case. It is up to the user to define the names and types of the test case attributes and record them in a parameter list. The checking tool uses this parameter list to process the test case table. It then checks whether all mandatory attributes are filled in, whether the test cases are classified by goal, purpose and type, whether they are automated and whether they have already been run. Since test cases refer to requirements or change requests, the tool also checks whether the referenced requirements actually exist. If they do, it creates a link between the test cases and the requirements. Inverting these links shows which test cases belong to which requirements, use cases and business rules. There should be no requirement and no use case without a test case. The tool documents every link between test cases and use cases and between test cases and requirements.

+----------------------------------------------------------+
|  S Y S T E M    C O N S I S T E N C Y    M E T R I C    R E P O R T  |
+----------------------------------------------------------+
| LANGUAGE: GERMAN/UML/C++/TCS     |    DATE: 22.06.13       |
| SYSTEM: LAGERHALTUNG             |    PAGE: 3 of 19        |
+----------------------------------------------------------+
|  E N T I T Y      C O U N T S                                 |
| Number of System TestCases                  ======>  356     |
| Number of Requirements to be tested         ======>  151     |
| Number of Requirements with TestCases       ======>  124     |
| Number of Code Components to be tested      ======>  229     |
| Number of Code Components with TestCases    ======>  80      |
+----------------------------------------------------------+
|  R E L A T I O N      C O U N T S                           |
| Number of TestCase/Requirement Relations    ======>  699     |
| Number of TestCase/Component Relations      ======>  10086   |
+----------------------------------------------------------+
|  D E F I C I E N C Y      C O U N T S                      |
| Number of Requirements with no TestCase     ======>  27      |
| Number of Code Components with no TestCase  ======>  149     |
+----------------------------------------------------------+
|  C O V E R A G E      M E T R I C S                        |
| Requirement Test Coverage Rate              ======>  0.821   |
| Code Component Test Coverage Rate           ======>  0.349   |
+----------------------------------------------------------+

Checking Consistency Across All Semantic Levels

To assess a software product as a whole, you have to combine the checking and measurement results of all its parts. A whole is only as good as the sum of its parts plus all the relationships between them.

Software systems are more than code. They consist of requirements documents, design models, code and test cases, and these layers have to be complete and consistent. So besides checking the individual artifacts, you also have to check that they are consistent with each other. It must be possible to trace code modules back to the architecture and requirements above them. That works through shared identifiers, through comments that point to requirement elements, or through the test cases. Test cases connect code modules to requirements: every requirement has a test case that confirms it is met, and it is met in the code. The quality of the overall system depends on how visible all internal and external dependencies are. Hence the effort to make these relationships visible.

Checking the consistency of software starts with the requirements documentation. Every functional requirement must be fulfilled by a use case and confirmed by at least one test case. Every use case must in turn be implemented by one or more code modules, and every code module or class must be assigned to at least one use case. Static analysis alone is enough to detect and record these links. Missing links can then be reported as quality deficiencies. Typical consistency deficiencies are:

• Requirements not fulfilled by any use case.
• Requirements not tested by any test case.
• Use cases that fulfill no requirement.
• Use cases not implemented by any code module.
• Code modules that cannot be assigned to any use case.
• Code modules without test cases.

The responsible tester or quality auditor gets a report on the missing relationships in the system and can point the developers to them without having to run a single test.

Why Software Metrics Matter

Completeness and consistency are one side of static software quality assurance, and they are established by comparing measurement results. The other side is conformity with general and product-specific quality requirements. These requirements are not only written down in requirements documents but also in the many publications on software development that record 50 years of experience in building software systems.

Painful experience with many bad solutions shows what a good solution should look like. The rules for a good solution are expressed as metrics that set a kind of target for the software. A metric is nothing more than a quantified rule of thumb (for example: software modules should be loosely coupled so each can evolve without affecting the others, or software requirements should be worded so precisely that no misunderstandings can arise). Tom McCabe’s much-cited cyclomatic number is ultimately just a rule of thumb for limiting control flow complexity. The same goes for other measures such as inheritance depth and class size. Whether such rules of thumb have been put into practice is best assessed with quantified measures.

The goal should be to bring the values from analyzing all types of software artifacts together in a central metrics database and compare them there. A database like this needs several import interfaces to take in values from the various analysis tools, and a graphical user interface that gives users different views of the numbers. The point is to express the quality of the software in figures. Sneed’s metrics database system has already proven itself for this purpose in several assessment projects. In them, software systems with up to three million code statements and 40,000 function points were checked and measured to draw conclusions for their further development.

Experience with Analytical Quality Assurance

Many IT decision-makers are skeptical about static software analysis. They don’t see the connection between the quality of a system’s construction and the quality of its behavior. So they focus their attention and their resources on what they do understand, empirical quality assurance: testing, in the spirit of “the proof of the pudding is in the eating”. What they overlook is that most problems with software behavior go back to how the software was built. Construction deficiencies often lead to behavioral failures, and it is far cheaper to uncover them through static analysis than in an elaborate test. Paul Duvall makes this point in his book “Continuous Integration”.

The other goal of analytical quality assurance, keeping the software fit for further development, is often ignored altogether. Software managers are too busy with today’s problems to look ahead.

What they miss is that most of today’s problems stem from short-sighted decisions in the past. If they invested more in securing system quality, they would have fewer problems maintaining and evolving their systems later.

The tools for analytical quality assurance have matured to the point where their cost hardly matters. It is minimal compared with the benefit these analyses bring. So analytical quality assurance can be expected to become more and more important. It will never fully replace testing, but it can uncover many quality deficiencies that testing would never bring to light.

Frequently Asked Questions

Can automated static analysis replace testing?

No. Malfunctions only become apparent when the software is running and its actual behavior is compared to its expected behavior. Static analysis, however, uncovers deficiencies that would never surface during testing, such as missing links between requirements, use cases, and code. Both approaches complement each other because many behavioral problems stem from design flaws.

Does every deficiency in the source code lead to a failure?

No. A deficiency is a violation of a quality rule, such as nesting code blocks more than three levels deep. Some deficiencies do indeed result in failures, such as missing exception handling or security checks. However, most types of deficiencies only hinder maintenance and further development, such as excessive coupling between code blocks or arbitrarily chosen data names.

Why are manual code inspections no longer sufficient in practice?

The effort required is out of proportion to the pace of development. In the late 1970s, Harry Sneed was responsible for the inspection of Nassi-Shneiderman diagrams and CHILL programs in the Siemens EWSD project and needed about half a day for a single module; the code inspection took over a year. Checking and measuring must therefore be automated, using objects taken directly from configuration management.

Can requirements written in natural language even be checked automatically?

Yes, provided the text has been tagged beforehand, either with XML tags or with keywords for actors, business rules, objects, use cases, and requirements. A text analyzer then checks the completeness and consistency of the references, such as use cases without a corresponding requirement or business rules that haven’t been implemented. Formulation rules can also be checked, for example, to ensure that every requirement statement contains a subject, object, and predicate.

What is the purpose of coding rules if experienced developers question them anyway?

They serve three purposes: consistent code, change-friendly code, and secure code. Objections from the team are useful in this context because they force us to rethink the relevance and purpose of individual rules. Ultimately, a development team must agree on a common set of rules. Rules regarding comments, indentation, and line length serve to improve readability, while rules regarding the number of parameters serve to improve testability.

How do you demonstrate that a refactoring has actually improved quality?

By conducting measurement before and after the refactoring, which requires quantifying quality, typically on a scale from 0 to 1. In the example described, the degree of modularity rose from 0.265 to 0.501 and testability from 0.304 to 0.548, while portability, reusability, and flexibility remained virtually unchanged.

Why should test cases also be reviewed and measured?

Because test cases are just as much a part of the software system as the requirements specification and the design model, even though they are delivered to customers only in exceptional cases. The assessment verifies whether all required attributes are present, whether the test cases are classified by goal, purpose, and type, automated, and executed, and whether the referenced requirements exist. No requirement and no use case should be left without a test case.

Why do many IT managers rely more on testing than on static analysis?

They fail to recognize the connection between the quality of the design and the quality of the behavior, and they direct their attention and budget toward what is familiar. What is overlooked is that design flaws often cause behavioral errors and are significantly cheaper to find through static analysis than through extensive testing. The second goal, lower costs for further development, is usually overlooked completely.