What is Regression Testing?
Regression testing is a type of testing that is done to verify that a code change in the software does not impact the existing functionality of the product. This is to make sure the product works fine with new functionality, bug fixes or any change in the existing feature. Previously executed test cases are re-executed in order to verify the impact of change.
Regression Testing is a Software Testing type in which test cases are re-executed in order to check whether the previous functionality of the application is working fine and the new changes have not introduced any new bugs.
This test can be performed on a new build when there is a significant change in the original functionality that too even in a single bug fix.
Regression means retesting the unchanged parts of the application.
Tutorials covered in this Series are:
What You Will Learn:
- Regression Test Overview
- When to perform this test?
- Can regression testing be performed manually?
- Automated Regression Testing Tools
- Why the Regression Test?
- Types of Regression Testing
- How Much Regression is Required?
- What do we do in Regression Check?
- Regression Testing Techniques
- How to Select a Regression Test Suite?
- How to Perform Regression Testing?
- Regression in Agile
- Regression Of GUI application
- Difference between Regression and Re-testing
- Regression Test Plan Template (TOC)
- Recommended Reading
Regression Test Overview
Regression test is like a verification method. Test cases are generally automated as test cases are required to execute again and again and running the same test cases again and again manually is time-consuming and tedious one too.
Example, Consider a product X, in which one of the functionality is to trigger confirmation, acceptance, and dispatched emails when Confirm, Accept and Dispatch buttons are clicked.
Some issue occurs in the confirmation email and in order to fix the same, some code changes are done. In this case, not only the Confirmation emails need to be tested but Acceptance and Dispatched emails also needs to be tested to ensure that the change in the code has not affected them.
Regression testing is not dependent on any programming language like Java, C++, C# etc. It is a testing method which is used to test the product for modifications or for any updates being done. It verifies that any modification in a product does not affect the existing modules of the product.
Verifying that the bugs are fixed and the newly added features have not created any problem in the previous working version of the software.
Testers perform functional testing when a new build is available for verification. The intent of this test is to verify the changes made in the existing functionality and the newly added functionality as well.
When this test is done, the tester should verify whether the existing functionality is working as expected and the new changes have not introduced any defect in functionality that was working before this change.
Regression test should be a part of the release cycle and must be considered in the test estimation.
When to perform this test?
Regression testing is usually performed after verification of changes or new functionality. But this is not the case always. For the release that is taking months to complete, regression tests must be incorporated in the daily test cycle. For weekly releases, regression tests can be performed when the functional testing is over for the changes.
Regression checking is a variation of retest (which is simply to repeat a test). When retesting, the reason can be anything. Say, you were testing a particular feature and it was the end of the day- you could not finish testing and had to stop the process without deciding if the test passed/failed. The next day when you come back, you perform the test once more – that means you are repeating a test you performed before. The simple act of repeating a test is a retest.
Regression test at its core is a retest of sorts. It is only for the special occasion that something in the application/code has changed. It might be code, design or anything at all that dictates the overall framework of the system.
A retest that is conducted in this situation to make sure that the said change has not made an impact on anything that was already working before is called Regression Test. The most common reasons why this might be conducted are because new versions of the code have been created (increase in scope/requirement) or bugs have been fixed.
Can regression testing be performed manually?
I was just teaching one of these days in my class, and a question came to me – “Can regression be done manually?”
I answered the question and we moved on in the class. Everything seemed OK, but somehow this question nagged me for quite a while later.
Over the many batches, this question comes multiple times in various different ways. Some of them are:
- To perform the test execution do we need a tool?
- How is regression testing performed?
- Even after an entire round of testing– newcomers find it difficult to discern what exactly regression test is?
And of course, the original question:
- Can this testing be performed manually?
To begin with, Test execution is a simple act of using your Test cases and performing those steps on the AUT, supplying the test data and comparing the result obtained on the AUT with the expected result mentioned in your test cases. Depending on the comparison result, we set the status of the test case pass/fail. Test execution is as simple as that, there are no special tools necessary for this process.
Automated Regression Testing Tools
Automated Regression Test is the testing area where we can automate most of the testing efforts. We run all the previously executed test cases on a new build.
This means that we have a test case set available and running these test cases manually is time-consuming. We know the expected results, so automating these test cases is time-saving and is an efficient regression test method. The extent of automation depends upon the number of test cases that are going to remain applicable over time.
If test cases are varying from time to time, the application scope goes on increasing and then automation of regression procedure will be the waste of time.
Most of the regression test tools are record and playback type. You will record the test cases by navigating through the AUT (application under test) and verify whether the expected results are coming or not.
#1) Ranorex Studio
Enhance Software Quality and maximize your resources with this powerful automated regression testing tool. You can execute more test cases in a fraction of the time with Ranorex which is up to 78% efficiency increase over manual testing.
- AdventNet QEngine
- Regression Tester
- Rational Functional Tester
Most of these are functional and regression test tools.
Recommended reading => Check here the list of top Regression Tools
Adding and updating regression test cases in an automation test suite is a cumbersome task. While selecting an automation tool for regression tests, you should check if the tool allows you to add or update the test cases easily.
In most cases, we need to update automated regression test cases frequently due to frequent changes in the system.
WATCH THE VIDEO:
For a more detailed explanation of the definition with an example, please check the following Regression Test video:
Why the Regression Test?
Regression is initiated when a programmer fixes any bug or adds a new code for new functionality to the system.
There can be many dependencies in the newly added and existing functionality.
It is a quality measure to check whether the new code complies with the old code so that the unmodified code is not getting affected. Most of the time the testing team has the task to check the last minute changes in the system. In such a situation, testing only affected application area is necessary to complete the testing process on time by covering all the major system aspects.
This test is very important when there is a continuous change/improvement added in the application. The new functionality should not negatively affect the existing tested code.
Regression is required to find the bugs that occurred because of a change in the code. If this testing is not done, the product might get critical issues in the live environment and that indeed can lead the customer into trouble.
While testing any online website, a tester reports an issue that the Price of the Product is not shown correctly i.e. it shows lesser price than the actual price of the Product, and it needs to be fixed soon.
Once the developer fixes the issue, it needs to be re-tested and Regression testing is also required as verifying the price at the reported page would have got corrected but it might be showing an incorrect price at the summary page where the total is shown along with the other charges or the mail sent to the customer still has the incorrect price.
Now, in this case, the customer will have to bear the loss if this testing is not performed as the site calculates the total cost with the incorrect price and the same price goes to a customer by email. Once the customer accepts, the Product is sold online at a lower price, it will be a loss for the customer.
So, this testing plays a big role and is very much required and important as well.
Types of Regression Testing
Given below are the various types of Regression :
- Unit Regression
- Partial Regression
- Complete Regression
#1) Unit Regression:
Unit Regression is done during the unit testing phase and code is tested in isolation i.e. any dependencies on the unit to be tested are blocked so that the unit can be tested individually without any discrepancy.
#2) Partial Regression:
Partial regression is done to verify that the code works fine even when the changes have been done in the code and that unit is integrated with the unchanged or already existing code.
#3) Complete Regression:
Complete Regression is done when a change in the code is done on a number of modules and also if the change impact of a change in any other module is uncertain. Product as a whole is regressed to check any changes because of the changed code.
How Much Regression is Required?
This depends upon the scope of newly added features.
If the scope of a fix or feature is too large, then the application area getting affected is also quite large and the testing should be performed thoroughly including all the application test cases. But this can be effectively decided when the tester gets input from a developer about the scope, nature, and the amount of change.
As these are repetitive tests, test cases can be automated so that a set of test cases alone can be easily executed on a new build.
Regression test cases need to be selected very carefully so that maximum functionality is covered in a minimum set of test cases. These set of test cases need continuous improvements for newly added functionality.
It becomes very difficult when the application scope is very huge and there are continuous increments or patches to the system. In such cases, selective tests need to be executed in order to save testing cost and time. These selective test cases are picked based on the enhancements done to the system and the parts where it can affect the most.
What do we do in Regression Check?
- Re-run the previously conducted tests
- Compare the current results with previously executed test results
This is a continuous process performed at various stages throughout the software testing lifecycle.
A best practice is to conduct a regression test after the sanity or smoke testing and at the end of functional testing for a short release.
In order to conduct effective testing, the regression test plan should be created. This plan should outline the regression testing strategy and the exit criteria. Performance testing is also a part of this test to make sure that the system performance is not affected due to the changes made in the system components.
Best practices: Run automated test cases every day in the evening so that any regression side effects can be fixed in the next day build. This way it reduces the release risk by covering almost all regression defects at an early stage rather than finding and fixing those at the end of the release cycle.
Regression Testing Techniques
Given below are the various techniques.
- Retest all
- Regression Test Selection
- Test case Prioritization
#1) Retest all:
As the name itself suggests, the entire test cases in the test suite are re-executed to ensure that there are no bugs that have occurred because of a change in the code. This is an expensive method as it requires more time and resources when compared to the other techniques.
#2) Regression Test Selection:
In this method, test cases are selected from the test suite to be re-executed. Not the entire suite is re-executed. Selection of test cases is done on the basis of code change in the module.
Test cases are divided into two categories, one is Reusable test cases and another one is obsolete test cases. The reusable test cases can be used in future regression cycles whereas obsolete ones are not used in the upcoming regression cycles.
#3) Test Case Prioritization:
Test cases with high Priority are executed first than the ones with medium and low priority. The priority of test case depends on its criticality and its impact on the product and also on the functionality of the product which is used more often.
The hybrid technique is a combination of Regression test selection and Test case Prioritization. Rather than selecting the entire test suite, select only the test cases which are re-executed depending on their priority.
How to Select a Regression Test Suite?
Most of the bugs found in the production environment occur because of the changes did or bugs fixed at the eleventh hour i.e. the changes done at a later stage. The bug fix at the last stage might create other issues/bugs in the Product. That’s why Regression checking is very important before releasing a Product.
Below is a list of test cases that can be used while performing this test:
- Functionalities which are frequently used.
- Test cases which cover the module where the changes have been done.
- Complex test cases.
- Integration test cases which include all the major components.
- Test cases for the core functionality or feature of the Product.
- Priority 1 and Priority 2 test cases should be included.
- Test cases which frequently fail or recent testing defects were found in the same.
How to Perform Regression Testing?
Now that we have established what regression means, it is apparent that it is testing also – simply repeating in a specific situation for a specific reason. Therefore, we can safely derive that the same method applies for testing in the first place can be applied to this too.
Therefore, if testing can be done manually then regression testing can be too. The use of a tool is not necessary. However, as time goes on applications get piled on with more and more functionality which keeps increasing the scope of regression. To make the most of the time, this testing is most often automated.
Given below are the various steps involved in performing this testing:
- Prepare a Test suite for Regression considering the points mentioned in “How to select Regression Test suite”?
- Automate all the test cases of the test suite.
- Update the Regression suite whenever it is required like if any new defect which is not covered in the test case is found, and a test case for the same should be updated in the test suite so that the testing is not missed for the same next time. The regression test suite should be managed properly by continuously updating the test cases.
- Execute the Regression test cases whenever there is any change in the code, the bug is fixed, new functionality is added, an enhancement to the existing functionality is done etc.
- Create a test execution Report which includes the Pass/Fails status of the executed test cases.
Let me explain this with an example. Please examine the below situation:
|Release 1 statistics|
|No. of requirements (scope)||10|
|no. of test cases/tests||100|
|no. of days it takes to develop||5|
|No. of days it takes to test||5|
|no. of testers||3|
|Release 2 statistics|
|No. of requirements (scope)||10+ 5 new requirements|
|no. of test cases/tests||100+ 50 new|
|no. of days it takes to develop||2.5 (since this half the amount of work than earlier)|
|No. of days it takes to test||5(for the existing 100 TCs) + 2.5 (for new requirements)|
|no. of testers||3|
|Release 3 statistics|
|No. of requirements (scope)||10+ 5 + 5 new requirements|
|no. of test cases/tests||100+ 50+ 50 new|
|no. of days it takes to develop||2.5 (since this half the amount of work than earlier)|
|No. of days it takes to test||7.5 (for the existing 150 TCs) + 2.5 (for new requirements)|
|no. of testers||3|
The following are the observations we can make from the above situation:
- As the releases grow the functionality grows
- The development time does not necessarily grow with releases, but the testing time does
- No company/its management will be ready to invest more time in testing and less for development
- We cannot even reduce the time it takes to test by increasing the test team size because more people means more money and new people also means lots of training and maybe also a compromise in quality as the new people might not be at par with the required knowledge levels immediately.
- The other alternative clearly is to reduce the amount of regression. But that could be risky for the software product.
For all these reasons, regression testing is a good candidate for automation testing, but it does not have to be done only that way.
Basic Steps to Perform Regression Tests:
Every time the software undergoes a change and a new version/release comes up, the following are the steps you can take to carry out this type of testing:
- Understand what kind of changes have been made to the software
- Analyze and determine what modules/parts of the software might be impacted – the development and BA teams can be instrumental in providing this information
- Take a look at your test cases and determine if you will have to do a full, partial or unit regression. Identify the ones that will fit your situation
- Schedule the time and test away!
Regression in Agile
Agile is an adaptive approach which follows an iterative and incremental method. The product is developed in short iterations called sprint which lasts for 2- 4 weeks. In agile, there is a number of iterations, hence this testing plays a significant role as the new functionality or code change is done in the iterations.
The regression test suite should be prepared from the initial phase and should be updated with each sprint.
In agile, Regression check is covered under two categories:
- Sprint Level Regression
- End to End Regression
#1) Sprint Level Regression :
Sprint level regression is done mainly for the new functionality or the enhancement that is done in the latest sprint. Test cases from the test suite are selected as per the newly added functionality or the enhancement that is done.
#2) End-to-End Regression :
End-to-End Regression includes all the test cases that are to be re-executed to test the complete product end to end by covering all the core functionalities of the Product.
As Agile has short sprints and it goes on, it is very much required to automate the test suite, the test cases are executed again and that too needs to be completed in a short span of time. Automating the test cases reduces the time of execution and defect slippage.
Given below are the various advantages of Regression test:
- It improves the quality of the Product.
- It ensures that any bug fix or enhancement that is done does not impact the existing functionality of the Product.
- Automation tools can be used for this testing.
- It makes sure that issues which are already fixed do not occur again.
Though there are several advantages, there are some disadvantages as well. They are:
- It has to be done for a small change in the code as well because even a small change in the code can create issues in the existing functionality.
- If in case automation is not used in the Project for this testing, it will be a time consuming and tedious task to execute the test cases again and again.
Regression Of GUI application
It is difficult to perform a GUI (Graphical User Interface) regression test when the GUI structure is modified. The test cases written on old GUI either become obsolete or need to be modified.
Re-using the regression test cases means GUI test cases are modified according to the new GUI. But this task becomes a cumbersome one if you have a large set of GUI test cases.
Difference between Regression and Re-testing
Re-testing is done for the test cases which fail during the execution and the bug raised for the same has been fixed whereas Regression check is not limited to the bug fix as it covers other test cases as well to ensure that the bug fix has not impacted any other functionality of the Product.
Regression Test Plan Template (TOC)
1. Document History
3. Regression Test Plan
3.3. Test Strategy
3.4. Feature to be tested
3.5. Resource Requirement
3.5.1. Hardware Requirement
3.5.2. Software Requirement
3.6. Test Schedule
3.7. Change Request
3.8. Entry /exit criteria
3.8.1. Entry Criteria for this testing
3.8.2. Exit Criteria for this testing
3.10. Test Cases
3.11. Risk /Assumptions
Let's take a look at each of them in detail:
1. Document History
Document history consists of a record of the first draft and all the updated ones in the below-given format.
|2||DD/MM/YY||ABC||Updated for the added feature|
References column keep a track of all the reference documents used or required for the Project while creating a test plan.
|1||SRS document||Shared drive|
3. Regression Test Plan
This document describes the change/update/enhancement in the Product to be tested and the approach used for this testing. All the code changes, enhancements, updates, added features are outlined to be tested. Test cases used for unit testing and Integration testing can be used to create a test suite for Regression.
Purpose of Regression Test Plan is to describe what exactly and how testing would be performed to accomplish the results. Regression check is done to ensure that no other functionality of the product is hampered because of the code change.
3.3. Test Strategy
Test Strategy describes the approach which will be used to perform this testing and that includes the technique that will be used, what will be the completion criteria, who will be performing which activity, who will write the test scripts, which regression tool will be used, steps to cover the risks like resource crunch, delay in production etc.
3.4. Features to be tested
Feature/components of the product to be tested are listed here. In regression, all the test cases are re-executed or the ones which affect the existing functionality are chosen depending on the fix/update or enhancement done.
3.5. Resource Requirement
3.5.1. Hardware Requirement:
Hardware Requirement is identified here like computers, laptop, Modems, Mac book, Smartphone etc.
3.5.2. Software Requirement:
Software Requirement is identified like which Operating system and browsers will be required.
3.6. Test Schedule
Test schedule defines the estimated time for performing the testing activities.
Example: How many resources will perform a testing activity and that too in how much time?
3.7. Change Request
CR details are mentioned for which Regression would be performed.
|S.No||CR Description||Regression Test Suite|
3.8. Entry/Exit Criteria
3.8.1. Entry Criteria for this testing:
Entry criteria for the Product to start Regression check are defined.
- Coding changes/enhancement/addition of new feature should be completed.
- Regression test Plan should be approved.
3.8.2. Exit Criteria for this testing:
Here the exit criteria for Regression are defined.
- Regression testing should be completed.
- Any new critical bugs found during this testing should be closed.
- Test Report should be ready.
3.9. Test Cases
Regression Test cases are defined here.
Any risk & assumptions are identified and a contingency plan is prepared for the same.
Tools to be used in the Project are identified. Such as:
- Automation tool
- Bug Reporting tool
Names and Designation of the people are listed here:
Regression testing is one of the important aspects as it helps to deliver a quality product by making sure that any change in the code whether it’s small or large does not affect the existing or old functionality.
A lot of automation tools are available for automating the regression test cases, however, a tool should be selected as per the Project requirement. A tool should have the ability to update the test suite as the Regression test suite needs to be updated frequently.
With that, we wrap this topic up and hope there will be much better clarity on the subject from now on.
Please let us know your Regression related questions and comments. How did you tackle your regression testing tasks?