How Do You Test a Legacy System Migration?
Migration testing checks whether a migrated system does exactly what the legacy system did before. A migration test rarely has a requirements specification or a design model to test against, so requirements-based and model-based test approaches don’t fit. The legacy system takes their place as the test oracle: what it does with a given input is the expected result. The test criterion is complete functional equivalence between the legacy and the migrated system.
In practice, testers record how the old system behaves in production, replay the same transactions against the new system and compare the outputs automatically: screen by screen, table by table, report by report. Code coverage shows when enough has been tested. If data moves along with the code, the affected data structures are measured as well, and the contents of the old and new databases are compared.
I described this method together with Harry Sneed in 2010, drawing on projects in which legacy systems written in languages like COBOL or PL/1 moved to new platforms. The tools named here belong to that period, while the principle of the legacy system as test oracle does not depend on any of them.
Why the Legacy System Replaces the Specification
Some projects allow test effort and duration to be calculated up front, so the customer can get a fixed price for testing. That works when the product can be tested against a requirements specification, in other words in a cleanly specified, stable development project. The requirements documents can be analyzed by hand to estimate effort and cost, or a text analysis tool can extract metrics such as function points or test points.
Maintenance and reengineering projects work differently. The software itself changes only slightly, so the test effort can be estimated from the difference between the original and the modified version.
Migration projects are a third type. Here the entire system is tested against the legacy system, and the source code of the predecessor is the basis for the effort estimate. Since no specification describes what the system is supposed to do, the behavior of the legacy system takes over that role. Detailed test cases come from what the old system does in daily operation, typically its online transactions and batch processes, and every deviation of the new system from that behavior is logged as a discrepancy.
Migration Testing in Seven Steps
A migration test is best built up step by step, and each step depends on the one before it.
- Measure the code. Measuring the source code that is going to be migrated shows what has to be tested and how much effort it takes.
- Plan the test. Planning sets the budget and the time frame and defines the individual work steps and the expected results.
- Design the test. Every test object the system uses has to be identified: user interfaces, database tables, system interfaces and reports. The design also fixes the test steps and their order.
- Generate the test data. The legacy system runs in a controlled environment while inputs, outputs and the current database states are recorded. Production data is anonymized.
- Run the migration test. Test cases are derived from the online transactions and batch processes and run automatically against the migrated system.
- Evaluate the results. The results of both systems are compared to check whether the migrated system delivers the same functionality and performance as the legacy system. Every deviation is logged as a discrepancy.
- Measure the coverage. Test coverage is measured against the targets agreed in the test plan. These targets (coverage of transactions, code, data and so on) form the basis of the test service agreement (TSA).
Measuring the Code Before Planning
“You can’t control what you can’t measure.” Following this line from Tom DeMarco, planning a migration test starts with measuring what is going to be migrated: the code. It is the code that gets transformed into a different form during a migration. If a data set is migrated as well, the affected data structures have to be measured too.
Code can be measured in several dimensions: complexity of the control flow, data usage, module interactions, reusability, maintainability, conformity and others. Which of these to measure depends on the information the measurement is supposed to deliver. For the test, the number of code units, database tables and data interfaces matters most, because the test has to account for all of them to meet the functional requirements.
Static analysis extracts the required metrics from the source code, the database schemas, the user interface definitions and the system interfaces. The results show how many data interfaces and how many database tables are needed to test each defined component. A static analysis tool that calculates the testability of the system on a scale from 0 to 1 can help here.
In past projects, measuring the source code usually took no more than a week, provided the right measurement tools were used. There are tools that handle new as well as old programming languages. That matters, because in most migration projects the predecessor system is written in an old language such as PL/1, COBOL or C.
What Goes into a Migration Test Plan?
A migration test plan covers the test objectives, the resources, an estimate of effort and duration, the test scope made up of test cases and test objects, and exit criteria that can be measured.
The test effort is measured by the number of test cases to execute and the number of test objects to validate. A test case can be derived from the usage profile of the production environment and defined as a single online transaction, but also as an event, an operation or a batch process. A test object, in turn, can be generated from production data and can be a user interface, a file, a database table, a system interface or a generated report.
Test cases and test objects together make up the test scope. Aligning that scope with the test productivity of past projects gives a rough estimate for the current migration test. It can be refined using the product type, the product’s testability score, the factors influencing the project and the test repetition factor.
The outcome of test planning is a test plan following ISO/IEC/IEEE 29119-3, the successor to IEEE 829, plus an estimate of the test effort including the test duration. With these two parameters, a price offer for the migration test can go to the client. The test objectives and the exit criteria deserve particular attention in that offer. The customer can only accept the test at the end if both are met, so they belong in the test plan in an easily measurable form, for example as figures on the test cases executed and the test objects validated.
How Do You Design a Test When the Documentation Is Missing?
Designing a migration test means deciding what to test and how. The “what” is described by test cases and test objects, which come from analyzing the current production process. Every relevant variant of an existing transaction is already a potential test case.
The documentation of the legacy system is no help here. It is rarely up to date and often missing entirely. The only way to design the test is for the tester to sit down with the system’s end users and record how they use it today. The result is retroactive documentation of how the system is used in operation. That written record also accounts for a large part of the effort for regression testing.
Test objects are easier to identify because they can be observed and measured. What matters most is the number of data elements each data object has: fields, columns, tags (markers that structure data or text) and widgets (components of a graphical user interface). The value ranges of these elements, which can be read from the production environment, are just as relevant.
The test design has to make clear where the test objects will come from and by what means the test cases can be recorded. It should exist in a structured, semi-formal format, for example a spreadsheet, a graphical tool or an XML document.
Where Does the Test Data for a Migration Come From?
Test data for a migration project comes from production. Unlike a development project, a migration project doesn’t require separate test data to be created. Copying the entire production environment is generally far too much effort, though. The test design has already identified individual transactions, events and batch runs that are representative, and these now run on the existing system in a controlled environment.
Before each test, snapshots of the user control panel, the input files and the database content are archived. After each test, new snapshots of the same objects are recorded in the same form, along with any output messages sent and reports printed. Personal data, such as customer data, is anonymized to meet data protection requirements.
All of this produces a large volume of test data, which in many earlier projects could only be handled with a dedicated server. Test tools help in two ways:
- to record user interfaces
- to manage the many old and new data snapshots that have to be kept
A capture-replay tool is another option. Where none was available, testers simply took screenshots of the user interface in the production environment and used them as the specification for testing the migrated system. With or without tool support, data acquisition in migration testing is mainly a matter of managing large volumes of recorded data.
Why Test Execution Has to Be Automated
What sets migration projects apart is how much of the work is testing. New systems are tested module by module, component by component, system by system over a long period. Because design, coding and testing are closely interwoven, it isn’t obvious how much total test effort goes into the project. Only the system test at the end stands out clearly as a pure test activity.
A migration project tests entire systems at the same time. The code conversion is usually automated and carried out by people who don’t know the system’s functionality, so there is no way to predict where errors will occur. If code is transformed incorrectly, these people usually won’t even notice. That makes it the testers’ job to prove that the migrated system is correct and to track down transformation errors. In the projects this method comes from, testing took 60 to 80 percent of the total effort.
The amount of input data for the test cases is considerable and hard to specify by hand, and so is the number of test cases in regression testing. Without automation, there is no way to finish the migration test in a reasonable time frame.
To find out afterwards which part of the migrated code was tested, probes go into every branch, or at least into every method or code block, before the test starts. Automation tools can place them. Once the test begins, the migrated system is bombarded step by step with data from the old transactions, with test drivers simulating the human user. If a transaction fails, the next one starts anyway. After each transaction or batch process, the contents of the output fields, the output messages and the new state of the affected database are recorded for later evaluation.
Validating all of this during the test run would take far too long. If no automated recording tool is available, the output screens should at least be captured manually as screenshots. The priority in this phase is to run the test with as few interruptions as possible, and anomalies are recorded for later review. There is no room for creative testing in a migration test.
Comparing Screens, Databases and Reports
Only once the migration test has finished are its results validated. The screen contents, interface data and database tables of the migrated system are compared automatically, one after another, with those recorded on the legacy system. If the formats differ, they are first converted into a common format.
Dedicated conversion and comparison tools do this. User interfaces that used to be mainframe memory images or UNIX screens are transformed into XML documents. The new user interfaces, such as web pages or mashups that combine existing content into something new, are converted into similar XML documents. That way, individual data elements can be compared selectively, regardless of where or how they appear on the screen.
Databases are handled the same way. The contents of the old databases (IMS, IDMS, IDS, ADABAS, DB2, VSAM, sequential files and so on) are downloaded just like those of the new relational databases and converted into CSV files. The two files are then compared column by column. Different formats or calculated values are checked against defined rules.
This database comparison is the part that data migration testing covers on its own: it checks whether the data arrived complete and correct. Migration testing goes further, because it also tests the converted code. The ISTQB Foundation Level syllabus lists migrations as a trigger for maintenance testing and names tests of data conversion for the case where data from another application is migrated into the system.
Reports and system interfaces may come in a completely different format than the old versions. The values of the individual data elements still have to be identical. If the old invoice amount was “EUR 19.99”, the new one has to show exactly that value, even if it sits in a completely different place on the new screen or printout. Tools solve this by extracting individual data elements from any printout, screen or text file and writing them to an XML file, with an XSD schema as the basis for comparison. This makes it possible to compare data from an old mainframe program printout with data in XML or WSDL format.
Comparing data at the level of these elementary units reveals even the smallest differences, such as a misplaced decimal point. When millions of data outputs have to be compared, automated comparison is the best way to find errors in the migrated code. The alternative is to compare the execution paths through the code, which requires tracing the tested transactions. There are suitable tools for that as well, such as TestDoc. It records not only the paths taken through the code but also how many times each method or procedure was executed.
How Do You Know When to Stop Testing a Migration?
The migration test can end when the measured coverage reaches the targets agreed in the test plan. Since the code is converted, not the functionality as such, the code is what has to be measured. Before the test runs, probes or tracking points are inserted into the code automatically. Whenever one of them is executed, it is written to a log file, and after the test run these log files show how effective the test was. If the coverage falls short, testing continues until the targets are reached.
Reports document different kinds of information. One shows test coverage as the percentage of probes executed and compares it with the coverage target from the service contract. Another documents the path of each test transaction, so it can be compared with the path in the original system to pinpoint where something went wrong. A third lists the code units affected by particular transactions. If a specific method or procedure needs a fix, this report tells which test cases have to be repeated.
A migration test turns up different types of errors than the test of a newly developed system. In new development, functions may be missing, data types defined incorrectly, business rules misinterpreted or results miscalculated. In a migrated system the errors are more subtle: decimal places get lost, data records are corrupted, data ends up in a different location, conditions are reversed. The results look right, but they aren’t. Since there is no way to know which statement was converted incorrectly, every statement has to be tested, and that means running the test repeatedly.
Besides coverage, the correctness of the output data and the number of defects actually found relative to the number predicted also count toward the decision to stop. The planning targets for both were set in the test agreement before the project started.
What Migration Testing Demands from a Team
Testing a migrated system is much closer to mass production than a classic system test: an enormous amount of converted code and data has to be tested almost blindly within a limited time frame. To demonstrate functional equivalence, ideally almost every converted code element is tested with a large data sample from production. That volume of data and the number of transactions, which run again and again, make a strong case for test automation.
Experience from migration projects shows that migration testing requires highly specialized testers and highly developed test tools. The degree of test automation matters more than the number of testers. Test organization is just as important, because a migration test has to be planned down to the smallest detail, and that takes experienced test managers. Few organizations have them.
Frequently Asked Questions
What is data migration testing?
Data migration testing checks whether data moved from an old system arrives complete and correct in the new one. The contents of the old and new databases are exported, converted into CSV files and compared column by column, with defined rules for different formats and calculated values. Migration testing goes further and also tests the converted code.
What does post-migration testing check?
It checks whether the migrated system delivers the same functionality and performance as the legacy system. Screens, interface data, database tables and reports from both systems are compared automatically, and every deviation is logged as a discrepancy. Coverage measurement then shows whether the executed test cases covered the converted code adequately.
Which tools does migration testing need?
Migration testing relies on tools in almost every step. Static analysis tools measure the code and the testability of the legacy system. Recording and capture-replay tools capture user interfaces and data snapshots, test drivers replay the old transactions, and conversion and comparison tools check screens, databases and reports. Tracing tools record execution paths.
How do you test interfaces during a migration?
System interfaces are identified as test objects in the test design, just like user interfaces, database tables and reports. During the test run, their output is recorded after each transaction. Afterwards the interface data is compared with the recording from the legacy system. If the formats differ, tools extract the individual data elements into XML for comparison.
