Data warehouse testing checks whether, after loading and transformation, the data warehouse holds exactly the data the transformation rules call for. Because there is no user interface where a tester could read off results, it needs a different test strategy: derive an assertion from every transformation rule and check the result values automatically. In the project described here, with around 5,300 rules, 4,700 could be translated into assertions automatically and checked against the loaded data within minutes.
Data Warehouse Testing: Why Classic UI Testing Falls Short
Conventional testing of business information systems has testers sitting at a workstation and operating the user interfaces. They supply the inputs, start a transaction and check the results. Each transaction is a test case. The same goes for transaction-oriented web systems. In this kind of system test, the tester simulates the future end user and needs no more qualification than the end user does.
Batch systems and background processes have always posed a different testing problem. Here you depend much more on test tools. The tester needs far more expertise, because most of the test is invisible. One or more input files are provided, then batch runs are started that process those files to update databases or generate reports. The tester’s job is to fill the input files and/or databases with test data, start the jobs and validate the results. That sounds simple at first, but it is actually a very hard task, much harder than testing a user interface. Data warehouse systems belong to this category of IT systems. When you test a user interface, the relationship between input data and result data is clear. They belong to the same transaction. And there are only a handful of them. An input screen holds around 10 data values. The output screen usually shows even fewer results. So per transaction, the tester deals with a manageable amount of data.
With a batch test, and even more so with a data warehouse, things look different. A data warehouse is usually fed with input files or tables from several source systems by load jobs. Each input medium has a large number of individual attributes, often more than a hundred. With several input sources, the number of data attributes multiplies. Inside the data warehouse system, many more jobs typically transform and aggregate these attributes. The link between the input data on one side and the data stored in the data warehouse on the other is nowhere near as clear as in UI testing, and it is hard to follow intuitively.
In this situation, the tester has to rely on formal means. He needs rules that describe how inputs relate to outputs. Otherwise, all he can check is whether the result data is plausible, that is, whether it falls within a valid range of values, regardless of what the input data looked like. That is a very superficial test. If every number > 0 and < 1000 counts as plausible, then every number between 0 and 1000 is a correct result, no matter how it came about. Such a test is better than none, but it is hardly enough to guarantee correctness.
Statistical testing, or comparing how often selected result types occur, falls short as well, because it can only reveal gross deviations between expected and actual values. It beats the plausibility check but still isn’t good enough to prove the results correct. At best, it uncovers a few sample deviations.
The only truly effective way to verify every individual result is to check each one against its rule. For each result value, you recalculate from the prescribed formula and the arguments or input values whether the value is correct. In principle, this check can be done by hand. The tester just needs one window on the input data, a second on the rule and a third on the results. Working this way, he needs a good half hour per result value. In larger data warehouse systems, a single source system can easily deliver around 5,000 attributes. Even if the tester verifies only one sample result value per result type, 5,000 result types add up to roughly 2,500 hours, or 312.5 days of intensive work, just to verify the results. Add further source systems and the transformations inside the data warehouse, and the manual effort multiplies.
The other problem with this manual comparison is that it is not only time-consuming but also very demanding. It takes the tester’s full concentration to trace the rule for every result type. Under today’s project conditions, you can hardly expect people to sustain that kind of mental effort. The error rate in tracing the rules would be higher than the error rate in coding them. That leaves only one solution: automate the results check. We gained our first experience with automated results checking in one of our early data warehouse test projects in 2006.
Specifying the Transformation Rules
Everything starts with the requirements specification. As Hutcheson puts it, without a specification there is no way to verify anything. In a data warehouse project, the specification is the sum of all data transformation rules. In this project, the data transformations were designed with an entity-relationship modeling tool, and the rules were specified in Excel spreadsheets. Each spreadsheet had one row per new data attribute or result type. Besides the table and attribute names, each row documented the attribute type, the attribute description and, of course, the rule for the attribute. The example described here uses values and rules from the load jobs of one source system. The other source systems and the transformation and aggregation jobs inside the data warehouse were tested following the same scheme.
Excerpt from the original rule table:
INTEREST_RATE: "Link to TAB_X / ATTR_D interest conditions applicable to the account."; If ATTR_B in TAB_Y(debit) has a value other than 0, this account is linked to an interest group. The following then applies: TAB_Y / KEY (Position 3-4) (Interest Type) A(2), TAB_Y / KEY (Position 5-9) (Interest Subtype) A(5). TAB_Y / KEY (Position 10-12) A(3). The above Key fields are concatenated in the ID. If in TAB_B the ATTR_D values are zeroes, the account interest condition has to be extracted from the X_TAB: TAB_X / ATTR_B (Base Rate) TAB_X / ATTR_S (Spread Rate). If both attributes <> 0 value, extract TAB_X If TAB_X / ATTR_D is available (i.e. <> 0), extract the same as for TAB_X_ID. If only ATTR_D of TAB_Y <> value, extract ATTR_D.
Analysts had written the rules in quasi-formal English prose. At first glance, automatically translating these informal rules into a formal, interpretable expression looked impossible. On closer inspection, though, certain basic patterns emerged. More than 4,700 of the 5,300 result attributes were specified using one of the following seven rule patterns:
- Rule class 1: A simple one-to-one assignment of a source attribute.
- Rule class 2: A simple assignment of a constant value.
- Rule class 3: A selection from a list of alternative constant values (enumeration).
- Rule class 4: A combination of several source attributes from one source table (concatenation).
- Rule class 5: A combination of several source attributes from several source tables (join).
- Rule class 6: An assignment of the target attribute based on an arithmetic operation with constant values and/or source attributes.
- Rule class 7: An assignment of the result of a function.
Each of these assignment rules could be conditional or unconditional. Conditional rules were tied to some condition, for example if, when, as long as, in case, etc. The condition operands could be constant values or source attributes. One special condition was the success of a join operation. Usually these were simple ifs with else/otherwise clauses, though they weren’t always unambiguous.
Naturally, different analysts phrased these basic rules a little differently, but they were similar enough to be recognized and classified. The constant values and source attributes were also easy to spot. Source attributes were always written in capital letters, and constant values were enclosed in single quotation marks. That was the basis for automatically converting these semi-formal rules into strictly formal, testable assertion statements, which an assertion compiler could translate into interpretable test procedures. Some rules first had to be rewritten to make them parsable. The testers took care of that. About 3,900 rules belonged to rule classes 1 and 2 and needed no revision. The remaining 1,400 rules had to be rewritten, but for lack of time, only 750 were converted, following these patterns:
Rule Class 3
Rules for a list of alternative values (enumeration) could consist of constants as well as variables. The operands were separated by ”!” characters.
assign ‘const01’ ! ‘const02’ ! ‘const03’
assign TAB_A.ATTR_1 ! TAB_A.ATTR_2 ! TAB_A.ATTR_3
Rule Class 4
Rules for combining several source attributes from one source table (concatenation) could consist of constants as well as variables. The operands were separated by ”÷” characters.
assign TAB_A.ATTR_1 ÷ ‘_‘ ÷ TAB_A.ATTR_2
Rule Class 5
Rules for combining several source attributes from different source tables were expressed as join operations.
assign join TAB_A.ATTR_1 with TAB_B.ATTR_2 with TAB_C.ATTR_3
Rule Class 6
Rules that assign target attributes through arithmetic operations had to be rephrased slightly to make them unambiguous.
assign TAB_A.ATTR_1 * -1 + TAB_B.ATTR_2 / 2
Since parentheses weren’t supported, the operands had to be put in the right order so they could be resolved from left to right. That was enough for most arithmetic expressions.
Rule Class 7
Rules that assign function results were formalized as a function call, with the parameters in curly braces.
assign Func.x {TAB_A.ATTR_1, ’2’, TAB_B.ATTR_2}
Finally, all conditions were brought into a uniform format. For the sake of readability, this format used an unbracketed list of and/or comparisons.
assign <assignment> if TAB_B.ATTR_1 = ’1’ and TAB_C.ATTR_2 = ‘0’
assign <assignment> if TAB_D.ATTR_3 < TAB_E.ATTR_1 or TAB_D.ATTR_3 > ‘0’
The outcome of a join operation could also be part of a condition, as the following example shows:
assign <assignment> if join TAB_A.ATTR_1 with TAB_B.ATTR_3 = true
Every if statement was tied to an assign statement. The assign statement came first, followed by the condition that governs it.
assign TAB_D.ATTR_2 if join TAB_D.ATTR_1 with TAB_B.ATTR_1 = true
The guiding principle behind this syntax was to stay as close as possible to the existing English prose rules, so the analysts responsible could understand them easily. The syntax is therefore a compromise between two needs: being understood by domain experts and being interpreted by a machine. As such, it comes close to a domain-specific language for data warehouse projects.
Excerpt from the adjusted rule table:
INTEREST_RATE: "Link to TAB_X.ATTR_D interest conditions.
Debit interest conditions applicable to the account.";
" ? assign TAB_Y.ATTR_D | TAB_Y.ATTR_C | TAB_X.ATTR_C | 'D' if TAB_Y.ATTR_D (debit) <> '0',
assign TAB_X.ATTR_C if TAB_X.ATTR_D <> '0',
assign TAB_X./ATTR_C | TAB_X.ATTR_D if TAB_X.ATTR_B <> '0',
assign '*na*' if TAB_Y.ATTR_D = '00' and TAB_X.ATTR_S = '0' and TAB_X.ATTR_B = '00'
assign TAB_X.ATTR_N|ATTR_Y|ATTR_O|ATTR_A if other.(comment).";
While formally revising the rules, the testers discovered documentation errors and reported them to the analysts. Most were incomplete rule descriptions (an if without an else, missing keys in join statements, etc.) that left room for interpretation and made an exact check of the rule impossible. This affected about 4% of all rules.
Turning the Rules into Test Scripts
Of the roughly 5,300 rules in the specification, more than 3,900 could be converted without any adjustment. That left 1,400 rules to adapt, about 600 of which turned out to be impossible to convert. In the end, just under 800 rules were rewritten in the formal syntax. That took 20 minutes per rule on average, so formalizing the rules took around 266 hours. The alternatives would have been a) not to test these rules at all or b) to write the assertion scripts by hand. Writing them by hand would have cost just as much effort, if not more. On top of that, two semantic levels with two different descriptions of one and the same rule would have had to be maintained. Code development has a similar problem. If code is generated from the specification, only the specification needs maintaining. But if the code is written by hand, as is usually the case, both the specification and the code must be kept up to date. Since that is too much effort in the long run, the specification soon gets neglected and only the code evolves. The analysts are out of the game, and the product stays in the hands of the programmers, with all the familiar problems of systems that have grown over time, above all that nobody understands them, not even the programmers themselves.
To avoid this, it makes sense to maintain just one description of an IT system: the one whose language is closest to human language, namely the domain-specific language. At the very least, the test scripts should be generated automatically. That is what happened in this project. A tool generated the assertions from the specified rules.
Generated test script:
file: ACCOUNT;
// This comparison procedure assumes that the old file contains the following attributes:
// from TAB_X Table: A_ID, A_TYPE, ATTR_S, ATTR_R, ATTR_C, ATTR_S, ATTR_D, ATTR_F, ATTR_N
// from TAB_Y Table: ATTR_C, ATTR_D, ATTR_E, ATTR_F, ATTR_P
// from TAB_Z Table: ATTR_R
if ( new.A_ID = old.ATTR_ID );
assert new.A_ID = old.A_TYPE if (old.A_TYPE = "G");
assert new.A_ID = "S" if (old.ATTR_A = "R" & old.ATTR_S = "S");
assert new.A_ID = "C" if (old.ATTR_A = "R" & old.ATTR_S = "C");
assert new.A_TYPE = "L" if (old.ATTR_A = "R" & old.ATTR_S = "L");
assert new.A_RATE = old.ATTR_C if (old.ATTR_B = "0");
assert new.A_INTEREST = old.ATTR_D;
assert new.A_LIQU = old.ATTR_E;
assert new.A_ACCT = old.ATTR_P;
assert new.A_PC = old.ATTR_PC;
assert new.START_DATE = "2005.05.15" if (old.ATTR_H <> "0");
assert new.REVIEW_DATE = old.ATTR_V if (old.ATTR_F <> "0" & old.ATTR_N <> "0") ;
assert new.REVIEW_DATE = "NULL" if (old.ATTR_F = "0") ;
assert new.PRIMARY_INDICATOR = "P" ! "S" ! "*n.a.*";
assert new.INDICATOR = "inactiv" if (old.ATTR_R = X"3") ;
assert new.INDICATOR = "inactiv" if (old.ATTR_R = X"1") ;
assert new.INDICATOR = "closed" if (old.ATTR_R = "C") ;
assert new.INDICATOR = "active" if (other);
end;What makes test assertions simpler than code is that, for a given file or table, they can be arranged in any order. Each attribute of a table is checked on its own, independently of the other target attributes. The only thing that matters is that all assertions for a table sit together in one test procedure. The test procedure carries the name of the target table and an if-key condition that links the target object to the source object. This if-key condition has the following format:
if (TAB_X.KEY_1 = TAB_A.KEY_1 & TAB_X.KEY_2 = TAB_B.KEY_2 & TAB_X.KEY_3 = TAB_C.KEY_3 &
The key comparison matches the target object with the correct source objects. Beforehand, the source objects’ data is pulled from the source tables with select and join statements and assembled into comparison objects that correspond to the target objects. In other words, for every target object there is a comparison object holding all the source data the target object is built from. Building these comparison objects from the source tables is a prerequisite for verifying the result values. All comparison objects of the same type, that is, all comparison objects for one target object type, are collected as rows in a table.
That is why the select statement for building the comparison objects is also generated as part of assertion generation. These select statements are derived from the join and select statements in the rule specification.
The assertion statements themselves are either conditional or unconditional. Conditional assertions are tied to an if condition.
The basic syntax of an assertion statement is:
assert new.TAB.ATTRIBUT <oper> <comparison expression>;
The assertion generator could not turn rules of rule class 7 (functions) into testable assertions, because the connection to the required function libraries was missing. These assignments were marked as comments in the assertion procedures, and the testers had to test them manually.
The assertion generator parsed the Excel spreadsheets containing the rules and converted the rules into assertions one by one. Each rule produced one or more assertions, and this took less than a minute for all 266 target tables. The output was 266 assertion procedures plus a log of all missing and untranslatable rules, statistics on the rules and SQL select statements for retrieving the source data for each target object. Apart from the syntactic correctness of the assertions, the biggest advantage of generating them automatically is that the process can be repeated. When the rules change, all you have to do is rerun the assertion generation to regenerate the affected assertion procedures.
Example 1: Statistics from the test script generation
+---------------------------------------------------+
| Assertion Generation completed !
| Number of E/R Rule Lines processed = 11384
| Number of new Tables processed = 266
| Number of new Attributes processed = 10910
| Number of old Tables processed = 132
| Number of old Attributes processed = 00891
| Number of new Attributes without a rule = 05573
| Number of Transformation Rules specified = 05337
| Number of Transformation Rules processed = 04742
| Number of Basel-II Rules recognized = 00890
| Number of Basel-II Rules processed = 00852
| Number of Complex Rules recognized = 00853
| Number of Complex Rules processed = 00562
| Number of Complex Basel Rules recognized = 00157
+---------------------------------------------------+
| Number of Assert Scripts generated = 00266
| Number of Assertions generated = 04897
| Number of Table Selectons generated = 00181
| Number of Assert Keys generated = 00023
| Number of Assert Conditions generated = 00308
| Number of Assert Concatenates generated = 00103
| Number of Assert Alternates generated = 00365
| Number of Assert Arithmetic generated = 00009
| Number of Test Cases generated = 05337
+---------------------------------------------------+One way to determine test coverage is the ratio of executed to planned test cases. To measure test coverage in this project and make test progress quantifiable, a separate test case had to be created for every rule to be tested. Since creating these test cases by hand would have taken a lot of effort, the assertion generator produced them as well, in the form of CSV files. These CSV files could later be imported into the test case management tool in full.
For each test case, the generated information included:
- A test case number that is unique across all rules.
- The test case description.
- The expected result, which consisted of the content of the rule.
- The description of the preconditions and postconditions.
- The priority of the test case. The assertion generator derived it from the complexity of the rule: rules of classes 1 and 2 were rated “Low,” classes 3 and 4 “Medium” and classes 5, 6 and 7 “High.”
- Various metadata, such as test case author, tester, test system, keywords and test case status.
- The test case group. To keep the number of test cases manageable, they were grouped by target table.
The project used Mercury’s Quality Center as its test case management tool. Its flexible import interface made it a good fit for the generated test cases.
Test Strategy for the Data Warehouse Test: The Test Process
To run the data warehouse test repeatedly with the tools described here, the team defined a repeatable test process. It had eight steps:
- Revise the data transformation rules in the original Excel spreadsheet.
- Generate the assertion procedures from the transformation rules.
- Generate the test data and the SQL select statements from the assertion procedures.
- Compile the assertion procedures.
- Read the source tables (source system) and save them as CSV files.
- Load the data warehouse tables.
- Read the target tables (data warehouse) and save them as CSV files.
- Compare the CSV files of the source and target tables automatically, using the assertions.
The output of these eight steps was a log of missing data objects and deviating data attributes, that is, data whose values did not match the result of the assertions. These were flagged as incorrect.
Verifying Test Results Against the Assertions
Functional testing requires an oracle. The oracle is the source of truth: it tells us whether the test results are right or wrong. James Bach writes: “An oracle is an evaluation tool that will tell you whether the system has passed or failed a test. In high volume automated testing, the oracle is probably another program that generates or checks the test results. The oracle is generally more trusted than the software under test, so a concern flagged by the oracle is worth spending time and effort to check…”
In this project, the oracle was an in-house tool called “DataTest,” which interpreted the assertion procedures and verified the test results against the rules. For each target table, the corresponding source data was collected and stored in an intermediate database, sorted by key. The tool then read the target table and verified each record against the source data belonging to it, checking every attribute against its assertions. Result data that deviated from the specified result, according to the assertion and the source data, went into a deviation log. The checks showed that it is entirely possible to validate a large volume of test results automatically within a few minutes.
In addition, the tool logged all target records without a matching source record and all source records without a matching target record. That gave the testers a list of every unpaired key and every incorrectly filled target attribute in the data warehouse. Naturally, the testers had to review these manually to determine whether each one was a real defect. The result was documented per record in the corresponding test case in Quality Center, and if the test case failed, a defect was reported. This made test progress measurable for controlling and reporting.
This approach uncovered quite a few incorrect assignments from the import files to the data warehouse tables. Unfortunately, not every attribute could be verified, because not every rule could be converted into an assertion. Of the 5,300 rules, only 4,700 could be transformed into assertions. The rest had to be checked by hand, which of course took a lot of time.
Example 2: Excerpt from the data validation report
+----------------------------------------------------------------------------------------+
| File/Table Comparison Report |
| Key Fields of Record(new,old) |
+----------------------------------------------------------------------------------------+
| New: ACCOUNT |
| Old: Attributes |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000USD114601001 | duplicate key in old File/Table |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000XXX104501001 | |
| New: ATTR_ID | G |
| Old: Constant_Value | L |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000YYY104501001 | |
| New: ATTR_C | XXX00 |
| Old: ATTR_C | 0 |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000ZZZ104501001 | |
| New: ATTR_D | 0 |
| Old: ATTR_P | 1 |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000ZZZ104501001 | |
| New: REVIEW_DATE | 9999-08-01_00:00:00 |
| Old: Constant_Value | NULL |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000YYY104501001 | |
| New: ATTR_P | GL |
| Old: Constant_Value | RE |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000XXX104601001 | |
| New: START_DATE | 9999-08-01_00:00:00 |
| Old: Constant_Value | NULL |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000XXX104701001 | |
| New: ATTR_X | G |
| Old: Constant_Value | C |
+----------------------------------------------+-----------------------------------------+
| RecKey:100000ATS196501100 | missing from the new File/Table |
+----------------------------------------------+-----------------------------------------+
| Total Number of old Records checked: 91 |
| Number of old Records found in new File: 08 |
| Number of old Records with duplicate Keys: 72 |
| Number of old Records not in new Table: 11 |
| Total Number of new Records checked: 59 |
| Number of new Records found in old File: 08 |
| Number of new Records with alternate Keys: 00 |
| Number of new Records not in old File: 51 |
| Total Number of Fields checked: 93 |
| Total Number of non-Matching Fields: 46 |
| Percentage of matching Fields: 51 % |
| Percentage of matching Records: 14 % |
+----------------------------------------------------------------------------------------+
More Automated Tests across the Data Warehouse
Beyond verifying the transformation rules, the project automated further test scenarios. All of them were developed in the scripting language Python, and over the course of the data warehouse project they grew into an extensive framework for data-driven test automation and static code analysis of the data warehouse environment. The test approaches built into this framework included:
- Quantity checks: Is the correct number of records passed between the individual stages?
- Default data: Are all default values used correctly?
- Data content: Does all data content match the specified and expected data types, data lengths, etc.?
- Code characteristics: Are the correct properties and flags used in the individual modules?
- Integrity check: Is the integrity of the loaded data ensured?
Here, too, automation made the test repeatable and, beyond that, made it possible to test the entire data warehouse in the first place.
Lessons from the Data Warehouse Project
The data warehouse test project led to the following conclusions:
- Verifying the content of a data warehouse depends on a precise and complete specification of the data transformation rules. It is the basis for checking the individual attributes. Without such a specification, efficient testing is impossible.
- In a mass test like this, the transformation rules must be convertible automatically. Otherwise the effort becomes far too high, which means you need a tool that generates assertions from the rules.
- The assertions must be machine-processable. Scanning the data attributes to see whether their content looks plausible is not enough. Each attribute must be checked automatically against its rule to establish whether it is correct.
- Systematic testing of a data warehouse system is impossible without powerful test tools, and those tools must be embedded in a well-designed test process. Process and tools have to complement each other.
- A test project this complex requires a separate test automation project, as Kaner describes.
The test approach described here became the basis for further data warehouse tests and was refined continuously along the way.
Frequently Asked Questions
Why is testing a data warehouse more complex than testing transaction-oriented systems?
A data warehouse is populated via load jobs from multiple source systems, and each input record often contains more than a hundred attributes. In an input form, about ten values have a clear relationship to the result of the same transaction. In a data warehouse, however, transformation and aggregation jobs distribute the values in such a way that the connection between the input and the result is hardly intuitive to understand.
Is it sufficient to check the result data from a data warehouse for plausibility?
No. A simple value range check accepts any number between 0 and 1,000 as correct, regardless of how it was derived. Even comparing the frequencies of selected result types reveals only rough deviations between target and actual values. Both approaches are better than no testing at all, but they do not prove the correctness of individual values. To do that, each value must be recalculated according to the rule by which it was generated.
Why is it not feasible to manually verify every single result value?
Verifying a single value takes a good half hour: one window for the input data, one for the rule, and one for the result. With 5,000 result types from a single source system, that adds up to about 2,500 hours. On top of that is the concentration required. The error rate when tracing the rules by hand would be higher than the error rate during coding.
Can transformation rules written in prose be evaluated by a machine?
For the most part, yes. Of 5,300 specified result attributes, over 4,700 followed one of seven rule patterns, including one-to-one mapping, selection from a list of constants, concatenation, joins, and arithmetic operations. Source attributes were consistently written in uppercase letters, and constants were enclosed in single quotation marks. Some of the rules had to be rewritten before translation.
What are the advantages of generating test scripts from the specification rather than writing them by hand?
Two descriptions of the same rule must both be maintained, and in most cases the specification falls into disuse while only the code survives. Generated scripts are syntactically correct and reproducible: if a rule changes, the generation process is restarted. In the project described, this approach produced 266 assertion procedures in less than a minute.
Does formalizing rules also reveal errors in the specification?
Yes. When rewriting rules into a formal syntax, documentation errors were found in about four percent of all rules, mostly incomplete descriptions: an if without an else, or missing keys in JOIN statements. Such gaps leave room for interpretation and prevent precise verification. The testers reported them back to the analysts, who corrected the rules.
Is there still manual testing work to be done despite automated result verification?
Yes. In the project described, 4,700 of the 5,300 rules could be converted into assertions; the remainder had to be checked manually. The generator could not translate function calls because the connection to the function libraries was missing; they were marked as comments. Testers also manually evaluated logged discrepancies and unpaired keys to identify genuine errors.
Besides tools, what else is required for a repeatable data warehouse test?
A defined process in which the tools are embedded. In the project described, it comprised eight steps: from revising the rules in the specification, through the generation and compilation of the assertion procedures, to reading the source and target tables and comparing them automatically. Without powerful tools and without a suitable process, systematic testing remains impossible.
