Why Web Service Testing Is Hard
For a long time, the biggest obstacle to using web services was the problem of testing them. Web service testing means simulating how the services are used: requests have to be generated automatically, and responses have to be validated automatically, quickly and reliably. To do this, Harry Sneed and Shihong Huang developed a tool called WSDLTest. It uses a test data description language built on pre-condition and post-condition assertions, both to generate WSDL requests and to validate WSDL responses.
WSDLTest is part of a larger tool set called DataTest for creating and processing system test data. This article presents the architecture and functionality of the tool and what its use in an e-government project showed.
I wrote this article together with Harry Sneed in 2007. It describes SOAP web services with WSDL interfaces. Today, interfaces are more often built as REST APIs, but the principle of generated requests and automatically checked responses still holds.
What Role Do Web Services Play in a Service-Oriented Architecture?
Web services are the reusable components of a service-oriented architecture (SOA). They became more and more important for IT business because companies wanted more flexible systems so they could react quickly to changes in their business environment. When a competitor comes out with a new marketing approach, a company has to be able to follow suit within a short time. The adaptability of IT systems has become vital to a company’s survival.
New laws add to the pressure. Regulations such as the Sarbanes-Oxley Act or Basel II have to be implemented within months, by a fixed date that is often close. Postponing them isn’t an option.
Under this kind of time pressure, it is no longer possible to plan and organize long-running projects. A working solution has to be designed and assembled within a limited time. Responding that fast requires reusable components that can be glued together within a standard framework to support a customer-specific business process. That framework is very often a service-oriented architecture, as offered by IBM, Oracle and SAP.
The components are the web services, and the business process on top of them can be defined in the Business Process Execution Language (BPEL). The glue that binds the business process to the web services, and the web services to each other, is the Web Services Description Language (WSDL).
The web service components themselves come from different sources. Some are bought, some are taken from the open source community, some are newly developed. Others come from existing software systems and are recycled for reuse in the new environment, which usually means they get wrapped.
Why Do Web Services Have to Be Tested?
Because nobody can guarantee that a web service component will work as expected, wherever it comes from. Even the ones you buy may not fit the task exactly, and if they aren’t compatible, serious interaction errors follow. Recycled components can be even worse. Legacy programs tend to contain many hidden errors that cancel each other out in a particular context. Move them into a different environment to perform a slightly different function, and the errors suddenly surface. The same can happen with open source components.
Perry and Kaiser showed that a component that is correct in one environment will not necessarily be correct in another. That’s why components have to be retested in every environment in which they are reused.
Nguyen addressed the specific problems of testing web applications. The distributed architecture, with its interplay of web browser, web server, data server, middleware, application server and so on, brings many new sources of potential errors. The many possible interactions between the components, combined with the many different parameters, increase the number of test cases needed, which in turn drives up the cost of testing. The only way to cope with these higher costs is test automation.
For services developed in-house, the reliability problem is the same as for any new software. They have to be tested extensively at every level: unit, component and finally system level. Sneed and Huang (Journal of Software Maintenance and Evolution 19(5), 2007) put the error rate of newly developed software at 3 to 6 errors per 1,000 statements, and these errors have to be found and removed before the software goes into production. A significant share of them comes from developers’ wrong assumptions about the task and the behavior of the environment. Such errors only come to light when testing happens in the target environment, with data produced by people who see the requirements from a different angle. That is the main reason for independent testers.
So web services should go through an independent test process, both individually and in combination with each other. That process should be well defined and supported by tools so that it is fast, thorough and transparent. Transparency matters especially when testing web services, because test cases have to be traceable and intermediate results open to inspection. Because of the volume of test data required, inputs have to be generated automatically and outputs validated automatically. Combinations of representative test data achieve high functional coverage, and comparing the results with the expected results ensures a high degree of correctness.
Which Tools Were Available for Testing Web Services?
When WSDLTest was developed, there was no shortage of tools for testing web services. The problem was their quality. Most of them were new developments that had not yet matured. They were hard to adapt to local conditions and required users to submit data through the web client’s user interface. Testing through the user interface is not the most effective way to test web services, as R. Martin pointed out in an article in IEEE Software. He suggested a test bus that bypasses the user interface and tests the services directly. That is the approach Sneed and Huang took.
A typical tool on the market at the time was Mercury’s QuickTest Professional. Users filled in and submitted a web page, and the tool then traced the request from the client workstation through the network by instrumenting the SOAP message. The message was traced to the web service that processed it; if that web service called another one, the tool followed that link too. The content of each WSDL interface was recorded in a trace file. This let the tester follow the path of a request through the architecture and examine the message content at different stages of processing.
Parasoft offered a similar solution, except that the requests weren’t started from a web client. They were generated from the business process procedures written in BPEL. This produced a larger volume of data and at the same time simulated production conditions, since most web service requests were expected to come from the BPEL scripts that control the business processes anyway. What the Parasoft solution lacked was a way to check the responses. They had to be inspected by eye.
One of the pioneers in web testing was Empirix. Its eTester tool let testers simulate business processes through the web clients. Their requests were recorded and translated into test scripts, which testers could modify to turn one request into many variants for a comprehensive functional test. The scripts were written in Visual Basic for Applications, so anyone familiar with VB could work with them. They could also verify the responses against the expected results. Unexpected results were sorted out and fed into the defect reporting system.
Other testing companies such as Software Research, Inc., Logica and Compuware were working on similar approaches at the time. It was clear that the market would soon be flooded with web service testing tools, and that it would take a while after that for them to reach the desired level of quality. Until then, Sneed and Huang saw room for custom solutions like the one described here. Graham and Fewster cover the various approaches to test automation.
The WSDLTest Approach
WSDLTest takes a somewhat different approach from the commercial tools. It is based on the schema of the WSDL description, which means it starts with a static analysis of the schema. Two objects are generated from this schema: a template for a service request and a test script. The test script lets the user manipulate the arguments in the request template and check the results in the web service response. The test driver is a separate tool that reads and sends the requests and receives and stores the responses.
Why the Tool Was Built
It’s often the circumstances of a project that lead to building a tool. Here, the project was testing an e-government website. The general user, the citizen, accessed the website through the standard web user interface. The local authorities, however, had IT systems that also needed to access the website to get information from the central government database. So they were offered a web service interface with nine different services, each with its own request and response formats.
The user interface of the e-government website was tested manually by human testers who played out the behavior of potential users. For the web services, a tool was needed that simulated the user programs by generating typical requests automatically and submitting them to the web service. Since web service responses aren’t readily visible, they also had to be validated automatically. The motivation for the tool can be summed up as follows:
- Web services can’t be trusted by default, so they have to be tested intensively.
- All requests should be tested with all representative combinations of arguments.
- All responses should be validated with all representative result states.
- Checking the target functions requires WSDL requests with specific arguments.
- Checking correctness requires validating the WSDL responses against the expected results.
Building a Request Template
Every test is a test against something. There has to be a source for the test data and an oracle against which the results can be compared. In WSDLTest, the oracle is the WSDL schema. It is either generated automatically from the interface design or written by hand by the developer. As a third option, it can be derived from the BPEL process description.
However it is created, the schema defines the complex data types according to the rules of the XML Schema standard. Complex types can contain other complex types, so the data tree has nodes that occur once or several times. The schema then defines the base nodes, meaning the leaves of the tree, and their order. These are the actual parameters. Here is an excerpt from the type definitions of an e-government web service schema:
<definitions>
<types>
<schema>
<complexType name="getProfile">
<sequence>
<element name="DatWsRequest_1"/>
</sequence>
</complexType>
<complexType name="DatWsRequest">
<sequence>
<element name="memoID" type="string" nillable="true"/>
</sequence>
</complexType>
</schema>
</types>
<message name="DatWebService_getProfile">
<part name="parameters" element="getProfile"/>
</message>
<portType name="DatWebServiceInterface">
<operation name="getProfile">
<input message="Interface_getProfile"/>
<output message="getProfileResponse"/>
<fault name="DatWsException" message="DatWsException"/>
</operation>
</portType>
<binding name="DatWebServiceInterfaceBinding" type="tns:DatWebServiceInterface">
<soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"/>
</binding>
</definitions>The parameter description is followed by the message descriptions, which identify the name and parts of each message, whether request or response. Next come the port type definitions, which list each callable service operation with the names of its input and output messages. These names refer to the messages defined earlier, which in turn refer to the parameters. After the port types come the bindings, which describe the SOAP prototypes composed of service operations.
A WSDL schema is therefore a tree. The SOAP prototypes refer to the service operations, which refer to the logical messages, which refer to the parameters, which refer to the data types, and the data types can refer to each other. Parsing this tree is called tree walking. The parser picks a top node and follows it through all its branches, collecting the subordinate nodes on the way down. At the end of each branch, it finds the elementary data types such as integers, Booleans and strings.
WSDLTest goes one step further and assigns a set of representative values to each elementary data type. Integers, for example, get a range from 0 to 10,000, and strings get different character combinations. These value sets are stored in tables that the user can edit before the test data is generated. The complete structure of a WSDL interface therefore looks like this:
Web service interface
SOAP prototypes
service operations
logical messages
parameters
data types
elementary data types
representative valuesThe data generator walks the schema tree down to the elementary data types and picks a representative value for each type, chosen at random from the set of possible values. From these values, it builds an XML data group like this:
<validate>
<validate>
<DatWsRequest>
<memoID>Kat221</memoID>
<subCompID>ThisData</subCompID>
<functionID>4711</functionID>
<version>1</version>
</DatWsRequest>
</validate>
</validate>This produces a WSDL request file with sample data that is stored for later runs. At the same time, the tool creates a test script that lets the tester override the generated values: a template with the data element names and their values. By editing the script, the tester changes the values of the service request. Since a script is generated for the output data as well, the tester also gets a template for verifying the responses.
<complexType name="getProfile">
<complexType name="DatWsRequest">
<sequence>
<element name="memoID" type="string" nillable="true"/>
<element name="subCompID" type="string" nillable="true"/>
<element name="functionID" type="string" nillable="true"/>
<element name="version" type="string" nillable="true"/>
</sequence>
</complexType>
</complexType>
<getProfile>
<getProfile>
<DatWsRequest>
<memoID>XXXXXXXXX</memoID>
<subCompID>YYYYYYYYY</subCompID>
<functionID>ZZZZZZZZZ</functionID>
<version>ThisData</version>
</DatWsRequest>
</getProfile>
</getProfile>This is the result of the schema analysis, a WSDL template that serves as input for the final generation of test data. A WSDL request is a cascading structure. It starts with type definitions, which are referenced by the message definitions, which in turn are referenced by the input and output operations assigned to a port. Most of the code, such as the parts defining the ports and the SOAP containers, is purely technical and highly repetitive. That’s why it helps to have a template you can copy and adapt to each request.
Writing Pre-Condition Assertions
The test scripts for WSDLTest are sequences of pre-condition assertions that define possible states of a web service request. The same assertion language is also used to test other kinds of data, such as relational databases, XML files and text files. A state is a combination of given values for the data types in the WSDL interface definition. The values belong to individual data elements, but their assignment can be interdependent, so the tester can, for example, set a particular combination of values:
assert new.Account_Status = "y"
if(old.Account_Balance < "0");There are six assertion types for assigning test data:
- a reference to another existing data value in the same interface
- a constant value
- a set of alternative values
- a range of values
- a concatenated value
- a computed value
A reference points to another existing value, which must be within the same WSDL.
assert new.Account_Owner = old.Customer_Name;A constant is given as a literal in the statement. All literals are enclosed in quotation marks.
assert new.Account_Balance = "0";Alternative values are listed and separated by an “or” sign, ”!”.
assert new.Account_Status = "0" ! "1" ! "2" ! "3";The values are assigned in rotation, starting with 0. The first account gets status 0, the second status 1, the third status 2, and so on.
A range of values is used for boundary value analysis. It is defined by the lower and upper bounds of a numeric range.
assert new.Account_Status = ["1" : "5"];This assigns the values 0, 1, 2, 4, 5 and 6 in rotation. The first account gets status 0, the second 1, the third 2, the fourth 4, the fifth 5 and the sixth 6.
A concatenated value joins two or more existing values with two or more constants into a single string.
assert new.Account_Owner =
"Mr. " | Customer_Name | " from " | Customer_City;A computed value is an arithmetic expression whose arguments can be existing values or constants.
assert new.Account_Balance = old.Account_Balance / "2" + "1";Assignments can be conditional or unconditional. A conditional assignment carries a logical expression that compares a data variable of the WSDL interface with another variable of the same interface or with a constant.
assert new.Account_Owner = "Smith"
if(old.Account_Number = "100922" &
old.Account_Balance > "1000");The tester adapts the assertions in the generated script to create a representative test data profile, with equivalence partitions and boundary value analysis, and with ascending and descending value sequences. The goal is to vary the input data so that a wide range of representative requests gets tested. To do this, the tester has to know what the web service is supposed to do and assign the values accordingly.
file: DAT-WS;
if (object = "DatWsRequest");
assert new.memoID = old.memoID;
assert new.subCompID = old.subCompID;
assert new.functionID = "4711";
assert new.version = "1";
endObject;
if (object = "DatWSProfile");
assert new.attributeName = old.attributeName;
assert new.typ = 21 ! 22 ! 23;
assert new.value = "Sneed";
endObject;
if (object = "DatWsCheckAssertion");
assert new.memoID = "Kati";
assert new.version = "3";
assert new.notAfter = "ThisData";
assert new.notBefore = "ThatData";
endObject;
end;Overwriting the Template Data
Once the assertion scripts are available, the template of a web service request can be overwritten with the asserted data. That is the job of the XMLGen module. It matches the WSDL file generated by the WSDLGen module against the tester’s assertion script. The data names in the script are checked against the WSDL schema, and the assertions are compiled into symbol tables, with separate tables for constants, variables, assignments and conditions.
XMLGen then reads the WSDL file and replaces the data values with the values derived from the assertions. If an assertion refers to a constant, the constant replaces the existing value of the XML element with the matching name. If it refers to a variable, the value of the XML element with that variable name moves into the target element. Alternative values are assigned one after another until the last one is reached, then assignment starts again with the first. Range values are assigned as the boundary values plus and minus one.
<DatWsCheckAssertion>
<memoID>Kati</memoID>
<version>3</version>
<DatWsConditions>
<notAfter>ThisData</notAfter>
<notBefore>ThatData</notBefore>
</DatWsConditions>
</DatWsCheckAssertion>The result is a sequence of web service requests with different representative states, combining the originally generated data with the assigned data. By changing the assertions, the tester changes the states of the requests and so ensures maximum data coverage.
Invoking the Web Services
The generated requests reach the server through the WS test driver, a simulated BPEL process with a loop. The loop is controlled by a list of web services in the order in which they are to be called. The tester can edit the list to match the order to the test requirements.
From the list, the driver takes the name of the next web service, reads the generated WSDL file with that name and sends a series of requests to the service. In synchronous mode, it waits for each response before sending the next request. In asynchronous mode, it sends several requests until a wait command appears in the list. It then waits until all the requests it sent have been answered before moving on.
The responses are stored in separate response files so that a postprocessor can check them later. The driver neither creates requests nor checks responses. The preprocessor creates the requests, and the driver only sends them out. The postprocessor checks the responses, and the driver only stores them. This reduces the test driver to a simple dispatcher. BPEL procedures suit this role well, because they have everything needed to call web services within a predefined workflow.
Writing Post-Condition Assertions
To verify the responses, WSDLTest uses the same assertion language that builds the requests, only with the opposite meaning. Here the assertions don’t assign values. They compare the response data with the values of previous responses or with constants.
A comparison with another existing value refers to that value.
assert new.Account_Owner = old.Account_Owner;A comparison with a constant gives the value as a literal, enclosed in quotation marks as with assignments.
assert new.Account_Balance = "33.50";The alternate assertion checks whether the response data matches at least one value from a list.
assert new.Account_Status = "0" ! "1" ! "2" ! "3" ;An assertion can also check whether the response data lies within a range, for example between 100 and 500.
assert new.Account_Balance = ["100.00" : "500.00"];Computed values work as well. The expression is evaluated first, and then the response data is checked against it.
assert new.Account_Balance = old.Account_Balance - "50";As with the pre-conditions, post-conditions can be unconditional or conditional. A conditional one is qualified by a logical if expression that compares two variables in the response, or a variable in the response with a constant.
assert new.Account_Status = "3"
if(old.Account_Balance < "1000");Whenever an assertion is not true, the tool records an error message with both the expected and the actual value. If there is an assertion for every attribute of the WSDL response, the response is 100 % verified. That isn’t required, though. The tester can limit the check to critical variables, and data coverage drops accordingly. Data coverage is measured as the number of verified results relative to the total number of results.
Checking the Results with XMLVal
The XMLVal module verifies the results of the web service. First, it translates the post-condition assertions into internal tables with variable references, constants, enumerations and ranges. In the process, it checks the data names and types against those declared in the WSDL schema to keep them consistent.
+-----------------------------------------------------------+
| WSDL Response Validation Report |
| File: DAT-WS.wsdl Params: Y Y Y Y |
| Object: DatWsCheckAssertion Date: 26.02.06 |
| Type : WSDL System: WebService |
| |
| Key Fields of Record(new,old) |
+-----------------------------------------------------------+
| New:DatWsCheckAssertion |
| Old:DatWsCheckAssertion |
+-----------------------------------------------------------+
| Non-Matching Fields | Non-Matching Values |
+------------------------+----------------------------------+
| RecKey:27013 | |
| New: memoId | Marta |
| Old: memoId | Kati |
+------------------------+----------------------------------+
| RecKey:27022 | |
| New: Version | 2 |
| Old: Version | 1 |
+-----------------------------------------------------------+
| Total Number of old Responses checked: 10 |
| Number of old Responses found in new File: 10 |
| Number of old Responses not in new File: 00 |
| Number of new Responses found in old File: 10 |
| Number of new Responses not in old File: 00 |
| Total Number of Attributes checked: 70 |
| Total Number of non-Matching Attributes: 07 |
| Percentage of matching Attributes: 90 % |
| Percentage of matching Responses: 100 % |
+-----------------------------------------------------------+
With the compiled tables, the tool then checks the web service response. First, it stores the expected values in a table with a key for each object occurrence. Second, it parses the WSDL result file and matches the objects there against the objects in the assertion tables. When it finds a match, it extracts that object’s attributes and compares their values with the expected ones. If they don’t meet the verification condition, the data names and values go into a list of non-matching results. It is then up to the tester to find out why the results don’t match. Besides listing the assertion violations, XMLVal also produces statistics on the degree of data coverage and the degree of correctness.
How WSDLTest Is Built
WSDLTest was put together from several components:
- GUI Shell
- XML File Writer
- XML File Reader
- Assertion Compiler
- XSD Tree Walker
- WSDL Analyzer
- Table Processor
- Error Handler
- Random Request Data Generator
- Selective Request Generator
- Request Validator
- Validation Report Generator
- WS Test Driver
Some of these components were taken from an existing tool, others were newly developed. The XML processing concept has been published and can be reused. The assertion language and the assertion compiler came from an earlier tool and were extended. This made it possible to get a prototype running in less than a month. Later, the tool was refined and extended. In its original form, it was built like this:
- The shell is implemented in Borland Delphi.
- The core is implemented in Borland C++.
- The tool uses no database, only temporary work files.
- It was designed to run on MS Windows or an equivalent environment.
- Shell and core are connected through an XML parameter file.
The User Interface
The Windows interface of WSDLTest takes parameters from the user, selects files from a directory and calls the back-end processes. Files can be selected from three directories:
- the assertion directory with the assertion text files
- the old file directory with the test inputs (csv, sql, xml and wsdl files)
- the new file directory with the test outputs (csv, xml and wsdl files)
There is also an output directory where the logs and reports are collected. The file names must be the same in all four directories, because that’s how the files are linked to each other. Only the extension may differ. So the actual response “Message.wsdl” is compared with the expected response “Message.wsdl” using the assertion script “Message.asr”, which produces the report “Message.rep”. Assertion scripts have the extension .asr. The names of all files belonging to a project are displayed with their types so the user can select them.
The Assertion Compiler
The assertion compiler reads the assertion scripts and translates them into internal tables that are interpreted at test time. Nine tables are generated for each file or object:
- a header table with information about the test object
- a key table with up to 10 keys used to relate the old and new files
- an assertion table with up to 80 assertion statements
- a condition table with an entry for each pre-condition to be met
- a constant table with an entry for each constant value to be compared or generated
- an alternatives table with an entry for each alternative value an attribute can have
- a concatenation table with an entry for each concatenated value
- a computation table with the operands and operators of the arithmetic expressions
- a replacement table with up to 20 fields whose values can be replaced by other values
Assertions have to be compiled before a file can be generated or validated. Many assertion scripts can be compiled at once before generation and validation start. The compiler writes its results to a log file, which the user sees at the end of each compilation run.
The WSDL Request Generator
The request generator consists of three components:
- schema preprocessor
- random request generator
- data assigner
XML schemas are usually generated by a tool that writes tags with many attributes together on one line. The schema preprocessor splits these long lines into many short, indented lines. That makes the XML text easier to read and parse and simplifies the work of the components that follow.
The random request generator then produces a series of WSDL requests with constant values as data. Which constant an element gets depends on its type. String data comes from representative strings, numeric data from adding or subtracting constant intervals, and date and time values from the clock. Within a request, data can appear as a variable between the tags of an element or as an attribute of an element. Attributes have no types, though. By definition they are strings, so they only get string values. The end result is a request in XML format with randomly assigned elements and attributes.
The data assigner reads in the randomly generated WSDL requests and writes them back out with adjusted data variables. In the assertion scripts, the user assigns specific arguments to the request elements by name, and compiling the scripts stores these names and arguments in symbol tables. For each data element of the request, the data assigner looks up the symbol tables to see whether an assertion value or a set of values has been assigned to it. If so, the asserted value overwrites the random value. If not, the random value stays. The WSDL requests thus become a mix of random values, representative values and boundary values, and the tester controls through the assertions which arguments reach the web service. The requests are stored in a temporary file for the request dispatcher. To tell them apart, each request gets a test case identifier as a unique key.
The WSDL Response Validator
The response validator consists of just two components:
- response preprocessor
- data checker
The response preprocessor works much like the schema preprocessor. It untangles the responses that come back from the web service so the data checker can process them more easily. It takes the responses from the test dispatcher’s queue file.
The data checker reads the responses and identifies the results by their tag or attribute name. It checks each name against the data names in the output assertion table. When it finds a match, it compares the value of that element or attribute with the asserted value. If the actual value differs, it reports a discrepancy in the response validation report. This is repeated for every response.
The test case identifier assigned to a request is passed on to its response, so each response is linked to a specific request. This lets the tester include the test case identifier in the assertions and compare responses by test case.
The WSDL Request Dispatcher
The request dispatcher is the only Java component. It takes the generated requests from the input queue file, wraps them in a SOAP envelope and sends them to the target web service. If there is a problem reaching the web service, it handles the exception. Otherwise, it simply records exceptions and moves on to the next request. It takes the responses out of the SOAP envelope and stores them in the output queue file, where the test case identifier identifies them uniquely.
The dispatcher also records when each test case is sent and when the response comes back. These are the start and end times of the test case. If the web service methods are instrumented with probes that record when they run, specific methods can be linked to specific test cases. This makes it easier to locate errors in the web services. It also shows which methods a request passes through on its way through one or more web services. Sneed and Huang saw this as a topic for further research.
What Did the E-Government Project Show?
The experience with WSDLTest in the e-government project was encouraging. The tool was still under development but already in use. Nine web services were tested, with an average of 22 requests per service. Of 47 verified responses, 19 contained at least one incorrect result. Of the more than 450 errors found in the project as a whole, the web service test uncovered about 23.
It’s hard to say how many of these errors testing in some other way would have found. To judge the efficiency of one approach, you would have to compare it with another. What can be said is that the test was relatively cheap and took no more than two weeks of tester effort.
The tool works for web service testing as long as the WSDL interfaces aren’t too complex. If they are, writing the assertions becomes too difficult and mistakes creep in. At that point, the tester can’t be sure whether an observed error was caused by the web service or by a badly formulated assertion.
Sneed and Huang cite a similar experience from the U.S. ballistic missile defense project about 30 years earlier, where about 40 % of the reported errors turned out to be errors in the test procedures. The test technology the RXVP test lab used in that project shows how little the basic methods have changed since the 1970s: setting pre-conditions, checking post-conditions, instrumenting the software, monitoring test paths and measuring test coverage. Only the environment has changed.
Open Problems for Testing Tools
Testing software systems is a complex task that is not yet fully understood. It touches many topics, such as:
- specifying test cases
- generating test data
- monitoring test execution
- measuring test coverage
- validating test results
- tracking system errors, etc.
WSDLTest addresses only two of these: generating test data and validating test results. Another tool, TextAnalyzer, analyzes requirements documents to extract functional and non-functional test cases, which are then stored in a test case database. The next step would be to use these abstract test cases to generate the pre-condition and post-condition assertions, closing the gap between the requirements specification and the test specification. The biggest obstacle is how informal requirements specifications are. Formal, detailed expressions have to be derived from an abstract, informal description. The model-driven development community faces the same problem.
Another direction is test monitoring, that is, tracing the path of a web service request through the system. The tool for that is TestDocu, also by Sneed and Huang. It instruments the server components to record which request they are working on. A trace file then shows the execution order of the web services, since many services are often involved in handling a single request. The biggest remaining task is to integrate these testing tools so they can be used as a whole rather than one by one. That means building a generic test framework with a common ontology and standardized interfaces between the tools.
There is still the fundamental question of how far testing should be automated. Perhaps it isn’t wise to automate the entire web testing process, and better to rely on the skills and creativity of human testers instead, because automation tends to hide important problems. Test automation versus creative testing remains an important topic in the testing literature.
Frequently Asked Questions
How do you test web services?
You simulate how they are used: generate requests automatically, send them straight to the service, and check the responses automatically against expected values. Going through the user interface is not the most effective route. Testing should be independent and happen in the target environment, for each service alone and in combination, with representative data for high functional coverage.
How can test requests be generated from a WSDL schema?
A parser walks the schema tree from the SOAP prototypes through operations, messages, and parameters down to the elementary data types, and gives each type a set of representative values, such as integers from 0 to 10,000. The result is a request template plus a script the tester edits to replace generated values with targeted ones.
How are web service responses validated automatically?
With post-condition assertions written in the same language that builds the requests. Each assertion compares a response value with a constant, a previous response, a list of alternatives, a range, or a computed value. Every violation is logged with the expected and actual value, and the share of checked results gives the data coverage.
