Overview of Data Migration Testing:
It is quite often heard that an application is moved to a different server, the technology is changed, it is updated to the next version or moved to different database server etc.,
From the testing point of view, it all means that the application has to be tested thoroughly end-to-end along with migration from the existing system to the new system successfully.
Tutorials in this series:
System testing has to be performed in this case with all the data, which are used in an old application and the new data as well. Existing functionality needs to be verified along with the new/modified functionality.
Instead of just Migration Testing, it can also be termed as Data Migration Testing, where the entire data of the user will be migrated to a new system.
So, Migration testing includes testing with old data, new data or combination of the both, old features (unchanged features), and the new features.
Old application is usually termed as ‘legacy’ application. Along with new/upgraded application, it is also mandatory to keep testing legacy application until the new/upgraded ones become stable and consistent. Extensive migration test on new application will reveal the new issues that were not found in the legacy application.
What You Will Learn:
Migration Testing is a verification process of migration of the legacy system to the new system with minimal disruption/downtime, with data integrity and no loss of data, while ensuring that all the specified functional and non-functional aspects of the application are met post-migration.
Simple Representation of Migration System:
As we know, the application migration to a new system could be for various reasons, system consolidation, obsolete technology, optimization or any other reasons.
Hence while the System in Use needs to be migrated to a new system, it is essential to ensure the below points:
Hence in order to ensure a smooth migration of the live system by eliminating those defects, it is essential to carry out Migration Testing in the Lab.
This testing has its own importance and it plays a vital role when the data comes into the picture.
Technically, it is also required to be executed for the below purposes:
Testing has to be performed both before and after migration.
The different phases of Migration test to be carried out at the Test Lab can be classified as below.
In addition to the above, the following tests are also executed as part of entire Migration activity.
Before performing this Testing, it is essential for any Tester to clearly understand the below points:
Hence it is essential to do a thorough study of the old and the new system and then accordingly plan and design the test cases and test scenarios to be covered as part of above the phases of testing and prepare the testing strategy.
Designing the test strategy for migration include a set of activities to be performed and few aspects to be considered. This is to minimize the errors and risks that occur as a result of migration and to perform the migration testing effectively.
Activities in this Testing:
#1) Specialized team formation:
Form the testing team with the members having the required knowledge & experience and provide training related to the system that is being migrated.
#2) Business risk analysis, possible errors analysis:
Current business should not be hampered after migration and hence carry out ‘Business Risk Analysis’ meetings involving the right stakeholders (Test Manager, Business Analyst, Architects, Product Owners, Business Owner etc.,) and identify the risks and the implementable mitigations. The testing should include scenarios to uncover those risks and verify if proper mitigations have been implemented.
Conduct ‘Possible Error Analysis’ using appropriate ‘Error Guessing Approaches’ and then design tests around these errors to unearth them during testing.
#3) Migration scope analysis and identification:
Analyze the clear scope of the migration test as when and what needs to be tested.
#4) Identify the appropriate Tool for Migration:
While defining the strategy of this testing, automated or manual, identify the tools that are going to be used. E.g: Automated tool to compare source and destination data.
#5) Identify the appropriate Test Environment for Migration:
Identify separate environments for Pre and Post Migration environments to carry out any verification that is required as part of testing. Understand and document the technical aspects of the Legacy and New system of Migration, to ensure that the test environment is set up as per that.
#6) Migration Test Specification Document and review:
Prepare Migration Test Specification document which clearly describes the test approach, areas of testing, testing methods (automated, manual), testing methodology (black box, white box testing technique), Number of cycles of testing, schedule of testing, approach of creating data and using live data (sensitive info needs to be masked), test environment specification, testers qualification etc., and run a review session with the stakeholders.
#7) Production launch of the migrated system:
Analyze and document the to-do list for production migration and publish it well in advance
Given below are the various phases of Migration.
Before migrating the data, set of testing activities are performed as a part of Pre-Migration test phase. This is ignored or not considered in simpler applications. But when complex applications are to be migrated, the Pre-Migration activities are a must.
Below is the list of actions that are taken up during this phase:
‘Migration Guide’ which is prepared by the Migration team needs to be strictly followed to carry out the migration activity. Ideally, the migration activity begins with the data back up on the tape, so that, any time the legacy system can be restored.
Verifying the documentation part of ‘Migration Guide’ is also a part of data Migration Testing. Verify if the document is clear and easy to follow. All the scripts and steps must be documented correctly without any ambiguity. Any kind of documentation errors, miss matches in the order of execution of steps also need to be considered important so that they can be reported and fixed.
Migration scripts, guide and other information related to actual migration needs to be picked up from the version control repository for execution.
To note down the actual time taken for migration from the point of start of migration till successful restoration of the system, is one of the test cases to be executed and hence the ‘Time taken to migrate the system’ needs to be recorded in the final test report which will be delivered as part of Migration test results and this information will be useful during the production launch. The downtime recorded in the test environment is extrapolated to calculate the approximate downtime in the live system.
It is on the legacy system where the Migration activity will be carried out.
During this testing, all the components of the environment will usually be brought down and removed from the network to carry out the Migration activities. Hence it is necessary to note the ‘Downtime’ required for Migration test. Ideally, it will be the same as that of the Migration time.
Generally, Migration activity defined in the ‘Migration Guide’ document includes:
It is advisable for the testers to verify the above in the backend of the system or by conducting white box testing.
Once the Migration activity specified in the guide is completed, all the servers are brought up and basic tests related to verification of successful migration will be done, which ensures that all the end to end systems are appropriately connected and all the components are talking to each other, DB is up and running, front end is communicating with the back end successfully. These tests need to be identified earlier and recorded in the Migration Test Specification document.
There are possibilities that the software supports multiple different platforms. In such case, Migration needs to be verified on each of these platforms separately.
Verification of Migration scripts will be a part of the Migration test. Sometimes individual migration script is also verified using ‘White box testing’ in a standalone testing environment.
Hence Migration testing will be a combination of both ‘white box and Black box testing’.
Once this migration related verification is done and corresponding tests are passed, the team can proceed further with the activity of Post-Migration testing.
Once the application is migrated successfully, Post-Migration testing comes into the picture.
Here end-to-end system testing is performed in the testing environment. Testers execute identified test cases, test scenarios, use cases with legacy data as well as a new set of data.
In addition to these, there are specific items to be verified in the migrated environments which are listed below:
All of these are documented as a test case and included in the ‘Test Specification’ document.
Since the scope of Post-Migration testing becomes very huge, it is ideal to segregate the important tests that need to be done first to qualify that Migration is successful and then to carry out the remaining later.
It is also advisable to automate the end to end functional test cases and other possible test cases so that the testing time can be reduced and the results would be available quickly.
Few tips for testers for writing the test cases for post-migration execution:
Migration of the system also calls for the testers to verify the ‘Backward Compatibility’, wherein the new system introduced is compatible with the old system (at least 2 previous versions) and ensures that it functions perfectly with those versions.
Backward compatibility is to ensure:
Hence it is essential to ensure the backward compatibility of the system by specifically carrying out the tests related to support backward compatibility. The tests related to backward compatibility needs to be designed and included in the Test Specification document for execution.
In case of any issues while carrying out the migration or if there is a migration failure at any point of time during migration, then it should be possible for the system to roll back to the legacy system and resume its function quickly without impacting the users and the functionality supported earlier.
So, in order to verify this, Migration failure test scenarios need to be designed as part of negative testing and rollback mechanism needs to be tested. Total time required to resume back to the legacy system also needs to be recorded and reported in the test results.
After rollback, the main functionality and the regression testing (automated) should be run to ensure that migration has not impacted anything and rollback is successful in bringing back the legacy system in place.
The test summary report should be produced after completing the testing and should cover the report on the summary of the various tests/scenarios carried out as part of various phases of migration with the result status (pass/fail) and the test logs.
Time recorded for the following activities should be clearly reported:
In addition to the above information, any observations /recommendations can also be reported.
Challenges faced in this testing are mainly with data. Below are few in the list:
#1) Data Quality:
We may find that the data used in the legacy application is of poor quality in the new/upgraded application. In such cases, data quality has to be improved to meet business standards.
Factors like assumptions, data conversions after migrations, data entered in the legacy application itself are invalid, poor data analysis etc. leads to poor data quality. This results in high operational costs, increased data integration risks, and deviation from the purpose of business.
#2) Data Mismatch:
Data migrated from the legacy to the new/upgraded application may be found mismatching in the new one. This may be due to the change in data type, format of data storage, the purpose for which the data is being used may be redefined.
This result in huge effort to modify the necessary changes to either correct the mismatched data or accept it and tweak to that purpose.
#3) Data Loss:
Data might be lost while migrating from the legacy to the new/upgraded application. This may be with mandatory fields or non-mandatory fields. If the data lost is for non-mandatory fields, then the record for it will still be valid and can be updated again.
But if the mandatory field’s data is lost, then the record itself becomes void and it cannot be retracted. This will result in huge data loss and should have to be retrieved either from the backup database or audit logs if captured correctly.
#4) Data Volume:
Huge Data that requires a lot of time to migrate within the downtime window of the migration activity. E.g: Scratch cards in Telecom industry, users on an Intelligent network platform etc., here the challenge is by the time, the legacy data is cleared, a huge new data will be created, which needs to be migrated again. Automation is the solution for huge data migration.
#5) Simulation of a real-time environment (with the actual data):
Simulation of a real-time environment in the testing lab is another real challenge, where testers get into different kind of issues with the real data and the real system, which is not faced during testing.
So, data sampling, replication of real environment, identification of volume of data involved in migration is quite important while carrying out data Migration Testing.
#6) Simulation of the volume of data:
Teams need to study the data in the live system very carefully and should come up with the typical analysis and sampling of the data.
E.g: users with age group below 10 years, 10-30 years etc., As far as possible, data from the live needs to be obtained, if not data creation needs to be done in the testing environment. Automated tools need to be used to create a large volume of data. Extrapolation, wherever applicable can be used, if the volume cannot be simulated.
Below given are few tips to be carried out in order to smoothen the data migration risks:
Hence considering the complexity involved in carrying out data Migration Testing, keeping in mind that a small miss in any aspect of verification during testing will lead to the risk of failure of migration at the production, it is very important to carry out careful and thorough study & analysis of the system before and after migration. Plan and design the effective migration strategy with the robust tools along with skilled and trained testers.
As we know that Migration has a huge impact on quality of the application, a good amount of effort must be put up by the entire team to verify the entire system in all aspects like functionality, performance, security, usability, availability, reliability, compatibility etc., which in turn will ensure successful ‘Migration Testing’.
‘Different types of Migrations’ that typically happen quite often in reality and the ways to handle their testing will be explained briefly in our next tutorial in this series.
About the Authors: This guide is written by STH Author Nandini. She is having 7+ years of experience into software testing. Also, thanks to STH Author Gayathri S. for reviewing and providing her valubale suggestions for improving this series. Gayathri is having 18+ years of experience in Software Development and Testing Services.
Let us know your comments/suggestions about this tutorial.