When you use third-party cloud services, cloud testing is mainly about checking whether a service fits your own requirements. The service test procedure does this in two phases: static analysis compares the requirements with the interface definition, and dynamic analysis runs the service with generated requests and checks the responses. An integration test of the whole follows.
Modern IT systems keep getting bigger and more complex. Building them once may be possible with professional help, but users are often overwhelmed when it comes to maintaining and evolving such complex systems. And the steady stream of new systems multiplies the maintenance problem. Software has to be corrected, changed, renovated and developed further all the time to keep pace with operational reality. The more new software gets built, the more capacity it takes to keep it alive. Users therefore need more and more staff, both to maintain the old systems and to build new applications.
Producing more and more code, faster and faster, is not a solution. The goal has to be gaining functionality without growing your own code base. The code mass has to be contained while still delivering the functionality that is needed. The only way to do that is to use third-party, ready-made building blocks that users don’t have to develop and maintain themselves. Responsibility for maintaining and evolving them lies with the provider. The rise of cloud computing promises a way out. In the cloud, users can combine the advantages of off-the-shelf software with those of in-house development. They don’t have to solve every detailed problem themselves. They use the standard solutions and focus their energy on the functions that are truly specific to their business.
Service-Oriented Requirements Analysis
Using cloud services requires service-oriented requirements analysis. It starts with modeling the business process to mark out the context of the application. Then comes the search for services that could help solve the problem.
You look for building blocks that can be used in the planned system. Only once the list of candidate building blocks is long enough do users start detailing their requirements, and they do so in a way that fits the available services. The functional and non-functional requirements set the minimum acceptance criteria. They describe only the functions that absolutely must be there and set the quality limits that absolutely must be met.
The requirements specification contains the following:
- Functional and non-functional requirements
- Business objects as a reference point for the data that has to be made available to the service
- Business rules as the desired processing logic of the selected services
- System actors as the subjects of the use cases
- Process triggers that start use cases
- Use cases
- Business interfaces
This document has to be machine-analyzable, because two things depend on it: comparing the processing steps of the use cases with the operations of the candidate services, and generating test cases for testing those services. Its main purpose is to serve as a test oracle. It therefore has to be comparable with the interface definition, which means it has to sit on the same semantic level. At the same time, it contains information that ties it upward to the business process. In that sense, the document links the higher-level business process with the lower-level services.
Cloud Testing: The Service Test Procedure
The test procedure has two phases:
- Static analysis
- Dynamic analysis
Static Analysis
Static analysis compares the content of the requirements with the content of the interface definition. A text analyzer scans the specification and builds tables of use cases, processing steps and interface data. It also builds a table of test cases. For every processing step, every action, every state and every rule, a matching test case is generated. The test cases are enriched with information from the interface definitions. In parallel, the interface schema is analyzed. Besides checking the rules and measuring the size, complexity and quality of the interfaces, this step builds tables as well. The tables from the specification analysis are then compared with those from the interface analysis. The processing steps of the use cases are paired with the operations, and the business interface data with the parameter data of the operations. Wherever a pair doesn’t match, an incompatibility is flagged.
If a service turns out to be incompatible with what the user has in mind, the user has four options:
- Reject the service and keep looking
- Adapt their own requirements to the service
- Build a wrapper around the service that adapts its results to their requirements
- Develop their own service
Users decide case by case which of these options fits best. What we want to avoid is users starting to develop their own services. That option should only be allowed as a last resort.
At the end of the static analysis we know, first, whether the service is a candidate at all based on its interface definition. To find out, we have measured its size, complexity and static quality. Second, we can compare the structure and content of the service interface definition with the structure and content we would like to have. If they are too far apart, there is no point even starting the second phase: the dynamic analysis.
Dynamic Analysis
In dynamic analysis, the service is executed and its results are recorded for comparison. The starting point is the interface schema and the test case table derived from the requirements specification.
The analysis has eight steps:
- The tester completes the test case tables with value assignments
- A test script is generated from the test case table and the interface schema
- The tester adds further preconditions and postconditions to the test scripts
- The test scripts are compiled and test objects are created
- Requests are generated from the interface schema and the preconditions of the test objects
- The requests are sent one after another and the corresponding responses are intercepted
- The responses are validated against the postconditions of the test objects
- The test metrics are evaluated
The test case table generated from the requirements specification is not complete. The tester still has to fill in the assignment of test values, for example. An automated component then merges the test case table with the interface definition into a structured test script. The tester can refine and extend the script.
From here on, everything runs automatically. The scripts produce test objects. The generator combines the interface schema with the preconditions to generate a series of requests for each test case. A test driver then sends the requests. It also receives the responses and stores them. The validator compares the responses with the postconditions and lists any deviating values in a defect report.
In the last step, the test metrics are aggregated and evaluated (test coverage, correctness, performance rating, etc.). With the test metrics report, the tester can judge to what extent the behavior of the service is suitable for the target application.
The test results serve as a basis for decisions, and for that they have to be presented in a form the decision-maker can easily understand and assess.
Summary
In the future, code building blocks will increasingly be offered as ready-made services that users only need to plug into their business processes. Programming, if the word even still applies, will happen in a higher-level process description language such as BPMN, S-BPM or BPEL. From there the interfaces are served and the individual services are called. Users no longer have to worry about the detailed implementation. They still have to test the building blocks they use, though: individually in a service unit test, as described here, and as a whole in an integration test. Either way, a paradigm shift is now due. We are moving from object-oriented to service-oriented software development. That should ease the maintenance problem raised at the start of this article considerably, even if it doesn’t solve it for good.
Frequently Asked Questions
Why Is It Not Enough to Simply Produce Software Faster?
More code means more maintenance work. Every new system must be corrected, modified, and further developed, which means it ties up staff resources on an ongoing basis. The solution lies in gaining functionality without increasing your own codebase: by using third-party, off-the-shelf building blocks, the maintenance of which remains with the provider. In 2012, cloud computing promised to make exactly this combination of standard solutions and in-house development possible.
Should requirements be fully specified before searching for suitable services?
No. The process begins with modeling the business process, followed by a search for services that can serve as building blocks. Only once there are enough candidates are the requirements detailed, in a way that aligns with the available services. Functional and non-functional requirements define only minimum acceptance criteria and quality thresholds.
Why does a requirements specification need to be machine-readable?
It serves as a test oracle. Only if a text analyzer can evaluate it can the processing steps of the use cases be mapped to the operations of the interface and test cases be generated automatically. To achieve this, the document must be on the same semantic level as the interface definition. At the same time, it connects the services upstream to the business process.
What should you do if a service does not match your requirements?
There are four options: reject the service and keep looking, adapt your own requirements to the service, build a wrapper that modifies the results, or develop a service yourself. Developing your own service should be the last resort, because it creates exactly the kind of code bloat you were trying to avoid.
Is it worth performing a static analysis of the interfaces before the service is executed?
Yes, it acts as a filter. Static analysis measures the size, complexity, and quality of the interface and maps processing steps to operations, as well as business interface data to parameter data. Where no pairing occurs, there is an incompatibility. If the requirement and the interface are too far apart, dynamic testing is unnecessary.
How much manual work remains in an automated service test?
Two steps remain manual: The tester assigns the test values in the generated test case table and adds additional preconditions and postconditions to the test script. After that, everything runs automatically: test objects, requests, sending via a test driver, validation of responses against the postconditions, and aggregation of the test metrics.
Do you even need to perform testing on third-party services yourself?
Yes. Even if the provider is responsible for the detailed implementation, the user must verify every component used: individually in a service unit test and during integration testing. The test results help determine whether the service’s behavior is suitable for the target application and must therefore be presented in a clear and understandable manner.
