This tutorial explains the Challenges of Automated Regression Testing. We will also learn about the Process and Steps to Automate Regression Testing:
We will explore how to automate the regression test cases starting from identifying them, selecting a tool to do the automation, doing cost, time & effort analysis, writing scripts and finally delivering it to the manual test team so that they can execute the test cases from anywhere anytime.
If you are wondering why only the regression test suite, its just that the regression test suite is the prime candidate for automation as it is a set of test cases which are repetitive and time taking. Thus, automating them would indeed save you lots of resources and it would be less time-consuming too.
You will get quick reports on the regression test cases and you can use these steps to automate any other test suites too.
=> Click Here For The Complete Regression Testing Series
Table of Contents:
Automated Regression Testing
Recently, when I wanted to start my new Automated Testing Project with four resources, I thought of applying any one of the Agile methodologies. But I was not able to continue because a series of questions were raised in my mind.
The questions were like:
- Is it possible to use Agile methodologies in Automated Testing?
- Can I use traditional tools?
- Should I go for open-source tools?
- What are the challenges I have to face if I am implementing automation in an Agile Environment?
In this tutorial we will analyze some of the challenges that we face while implementing Automation with Agile methodologies. Automated Regression Testing in an Agile environment stands as a risk of becoming chaotic, unstructured and uncontrolled.
Agile Projects present their own challenges to the Automation team. Agile methodology emphasizes team collaboration and frequent delivery of a product.
Factors like unclear project scope, Multiple iterations, Minimal documentation, early and frequent Automation needs and active stakeholder involvement, etc., demand a lot of challenges from the Automation Team.
Automated Testing: Challenges in an Agile Environment
There are several Agile challenges for the Automation team.
A few of them are briefed below.
Challenge 1: Requirement Phase
Test Automation developer captures the requirements in the form of “user stories”, which are brief descriptions of the customer-relevant functionality.
Each requirement has to be prioritized as follows:
High: These are mission-critical requirements that absolutely need to be done in the first release.
Medium: These are the requirements that are important, but can be worked around until implemented.
Low: These are the requirements that are nice-to-have but not critical to the operation of the software.
Once priories are established, the release “iterations” are planned. Normally, each Agile release iteration takes between 1 to 3 months to deliver. Customer/software folks take the liberty to make too many changes to the requirements.
Sometimes, these changes are so volatile, that the iterations are bumped off. These changes are the greater challenges in implementing the Agile Automation testing process.
Challenge 2: Selecting the Right Tools
Traditional, test-last tools with record-and-playback-features force teams to wait until after the software is done. Moreover, traditional test automation tools don’t work for an Agile context because they solve traditional problems that are different from the challenges facing Agile Automation teams.
Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. So the choice of testing tools becomes critical for reaping the efficiency and quality benefits of agile.
Challenge 3: Script Development Phase
Automation testers, developers, business analysts, and project stakeholders altogether contribute to kick-off meetings where “User-Stories” are selected for the next sprint. Once the “User-Stories” are selected for the sprint, they are used as the basis for a set of tests.
As the functionality grows with each iteration, Regression Testing must be performed to ensure that existing functionality has not been impacted by the introduction of new functionality in each iteration cycle.
The scale of the Regression Testing grows with each sprint and ensures that this remains a manageable task and the test team uses the test automation for the regression suite.
Challenge 4: Resource Management
The Agile approach requires a mixture of testing skills, that is, test resources will be required to define unclear scenarios and test cases, conduct Manual Testing alongside developers, write automated regression tests and execute automated regression packages.
As the project progresses, specialist skills will also be required to cover further test areas that might include integration and Performance Testing.
There should be an appropriate mix of domain specialists who plan and gather the requirements. The challenging part in Resource management is to find out test resources with multiple skills and allocate them.
Challenge 5: Communication
Good communication must exist among the Automation Testing team, developers, business analysts, and stakeholders. There must be a highly collaborative interaction between the client and the delivery teams.
More client involvement implies more suggestions or changes from the client. It implies more bandwidth for communication. The key challenge is that the process should be able to capture and effectively implement all the changes and the data integrity needs to be retained.
In traditional testing, developers and testers are like oil and water, but in an agile environment, the challenging task is that they both must work together to achieve the target.
Challenge 6: Daily Scrum Meeting
Daily Scrum Meeting is one of the key activities in the Agile Process. Teams do meet for 15 minute stand up sessions. What is the effectiveness of these meetings? How long have these meetings helped Automation Practice Developers? etc., are discussed at this meeting.
Challenge 7: Release Phase
The aim of the Agile project is to deliver a basic working product as quickly as possible and then to go through a process of continual improvement. This means that there is no single release phase for a product. The challenging part lies in integration testing and acceptance testing of the product.
Steps to Automate Regression Testing
The process to be followed to automate regression can be precisely divided into the following steps:
These 7 steps are explained below in detail in simple terms for your easy understanding.
1. Identification
#1) Identify the test cases which should be a part of the regression test suite.
- To start automating regression test cases, the very first thing that you need to do is to get the regression test cases identified and properly defined with all the steps, data and prerequisites.
- To make sure you have an effective regression test suite, don’t forget to include:
- Test cases with recurring defects.
- Test cases that cover end to end scenarios.
- Test cases that are more visible to the end-users.
- Test cases on boundary values.
- Good mix of positive and negative test cases.
- Complex test cases.
#2) Identify the automation tools that are best for your requirements and application behavior. Once the Regression test cases are identified and ready for automation, identify the tools that fit your test cases the best.
The best way to identify the tools is to make a matrix with the tools and your requirements and then keep a track of which tool satisfies which requirements.
Suggested Reading => List of the Top Automation Testing Tools
#3) Identify the Programming language that you want to use. With so many tools available on the market, multiple languages are supported by these tools. Thus, it’s important to identify the programming language in which you want to write your automation test cases.
Example: Let’s assume that we have a project where we want to automate a regression test suite for a browser-based application.
As Explained above, we will identify the Test cases.
- Let’s assume that our test case is “ Verify that a user can log in successfully using a valid username and password”.
Next, we will identify the Automation Tools.
- A browser-based test case can be automated with “Selenium”, “Ranorex”, “TestComplete”. Let’s decide on the Selenium tool as it best fits the requirements.
Now, let’s identify a programming language.
- We want to use “Java” as the programming language as it’s a highly supported language.
2. Analysis
#1) Do Cost analysis. It is very important to work within the budget limits. Thus, after the identification phase, you’ll have an idea of how many test cases need to be automated and which are the possible tools that can be used.
All the findings from the identification phase will help you to come up with an approximate budget and thus, you can get any approval, etc., if required.
#2) Do resource and effort analysis to see if you have the resources to work on this. Along with the cost analysis, it is very important to do a resource and effort analysis for better allocation of resources and using their time on the project efficiently.
While estimating resources and efforts, make sure to account for risks like, if someone falls sick, or if some test cases need more resources for execution, etc.
#3) Do Time Analysis. Time analysis is required to make sure that you can finish the automation project within the budget and timelines. While working on time analysis it will be helpful to prepare a timeline chart to monitor the progress during the development.
For a better timeline analysis of the project:
- Identify the tasks and subtasks for your projects.
- Prioritize tasks and subtasks.
- Draw a Gantt chart or network diagram for better picturization of the timeline.
Example: Going forward with our example in consideration from the identification phase, the explanation for this phase is given below:
Cost Analysis:
Let’s assume that the approximate cost for this project is $X amount.
Resources and Efforts:
For this, one test case has to be automated from end to end and we need one resource full time for approx 24 hours and we also need one more resource to review the work. Thus, we need 2 resources and the effort estimation is around 40 hours.
Time Analysis:
For this, let’s draw a small Gantt chart to see the timeline.
3. Training/Hiring
#1) If some resources require training, then plan their training. Sometimes you may have some manual testing resources who are interested in writing automation test cases or some people have worked on automation but different tools and are willing to learn the tool which you have selected for your automation.
Identify these resources and plan for their training so that they can start working on the automation of regression test cases.
#2) If we need more resources then work on the hiring plan. Once you have done the resource analysis for the efforts and if you are not able to satisfy the needs with the already available resources, then, plan on hiring some new resources with proper skills that are required for the project within the budget.
Example:
Let’s say we already have a resource who is familiar with Java concepts and wants to learn Selenium. Then we’ll arrange for selenium training for that person.
If we don’t have any resources available for automation. Then we’ll hire a person who has some background in automating such test cases using selenium and java.
4. Framework and Guidelines
#1) Once the tool and resources are ready, work on coming up with a framework or deciding on which one to use from the existing frameworks. Already built multiple frameworks can be used or you can build your framework from scratch.
When selecting or building a framework, make sure that you involve the components about, test cases, logs, reports, input, database connection, etc.
#2) Decide on the other supporting tools that you want to use. As developing automation scripts is a development task that involves writing code, it would be much better to figure out the other development tools which would be needed to support writing your scripts.
For Example, some tools that can be of help include Git, GitHub, Jenkins, etc.
#3) Outline the guidelines for writing automation scripts. We need to outline the guidelines so that all the resources working on the project are in sync and use the same naming conventions, the same procedures for code check-in/check-out, and the same programming language.
Example:
Framework: Let’s decide on a BDD (behavior-driven development) framework for automation testing.
Supporting tools: The tools that we need to fully support automation will be: GitHub, Jenkins, Log4J, Cucumber, and JUnit.
5. Automation Scripts
Start writing automation scripts. Once we have everything i.e., the tool, programming language, required skills and test cases that need to be automated, we can start writing automation scripts.
While writing the scripts, we have to make sure that:
- Guidelines are followed.
- We are using the tools.
- Test cases are modularized.
- We should be able to reuse the components if they are required in multiple test cases.
We also have to make sure that the code is properly maintained in the version control tool so that all the team members can collaborate easily.
Example:
Let’s write actual scripts to make this test case run. Samples from the scripts can be as shown below.
First, the cucumber scenario for this test case would look as below:
Feature: Verify Login functionality
As a user I want to Login into the application
Scenario Outline: Login into application
Given I open application
When I Enter username “<username>”
And I Enter password “<password>”
And I click on Login button
Then I go to Home page
Examples:
| username | password |
| testuser1 | password1|
| testuser2 | password2 |
After the feature file, we will implement the step definition for the steps mentioned in the feature file.
public class Login { LoginImpl loginImpl = new LoginImpl(); @Given("^I open application$") public void i_open_application() { loginImpl.openURL("URL"); } @When("^I Enter username \"([^\"]*)\"$") public void i_Enter_username(String arg1) { loginImpl.enterUserName(arg1); } @When("^I Enter password \"([^\"]*)\"$") public void i_Enter_password(String arg1) { loginImpl.enterPassword(arg1); } @When("^I click on Login button$") public void i_click_on_Login_button() { loginImpl.clickLoginButton(); } @Then("^I go to Home page$") public void i_go_to_Home_page() { loginImpl.verifyHomePage(); } }
Finally, the actual implementation of the login functionality class would look as below:
public class LoginImpl { WebDriver driver; public LoginImpl(){ System.setProperty("webdriver.chrome.driver", "webdriver/chromedriver.exe"); driver = new ChromeDriver(); } public void openURL(String string) { driver.get(string); } public void enterUserName(String arg1) { driver.findElement(By.id("UserName")).sendKeys(arg1); } public void enterPassword(String arg1) { driver.findElement(By.id("Password")).sendKeys(arg1); } public void clickLoginButton() { driver.findElement(By.id("LoginButton")).click(); } public void verifyHomePage() { String currUrl = driver.getCurrentUrl(); if(currUrl.equals("homePageURL")) { System.out.println("Home page verified"); } } }
6. Review
#1) Code review: Once an automation developer is done with writing automation scripts, it is very important to go through the different levels of code review.
Code reviews help in
- Identifying wrong verifications.
- Finding code optimization points.
- Finding better ways to implement the functionality for efficient use of resources.
Code reviews also expose a developer’s work to other developers and give it a different perspective and room for improvement too.
#2) Test cases review: Along with the code review, the test case review is also very important. We need to make sure that the automation test scripts when executed perform the same set of actions and verifications as we expect from the manual test cases.
Thus, reviewing automation test cases with business analysts or test experts helps in boosting the confidence in automation testing of those test cases.
Example:
For our example in consideration, let’s say for code review we got comments like “search element by id and not name”. Here, we’ll take this into account and modify our script accordingly.
Also, a test review can be done to add steps for testing if you are on the home page after a successful login. We will add this to our scripts too.
7. Deliver
Deliver test cases so that anyone can run them at any time. Once the automation scripts are ready for use, it is very important to come up with a delivery plan for the automation scripts.
This plan is required as we want to make sure that automating the test cases does not limit its execution to a specific set of people or skills. Everyone in the team or the project should be allowed to execute the test cases.
One possible delivery plan is to provide a Jenkins job that can be triggered to execute automated test cases.
Example:
In our case, let’s assume that we have delivered the test case using a Jenkins job. This Jenkins job, takes the code from GitHub, builds it and runs the test cases on a different machine.
Once the job is successful, it will show you the test report generated. Anyone who has access to Jenkins can run this job. This job can be scheduled to run at a specific time.
Conclusion
If we can meet these challenges in a well-optimized manner, then Automated Regression Testing in the Agile environment is an excellent opportunity for the QA to take up leadership of the agile processes.
It is better placed to bridge the gap between the users and developers, understand what is required, how it can be achieved and how it can be assured prior to deployment.
Automation practices should have a vested interest in both, the results, as well as continue to ensure that the whole evolving system meets business objectives and is fit for purpose.
Automating a regression test case is always helpful as it’s the best candidate to start automation testing. You can follow the steps mentioned above to automate any test suite and not just the regression.
Automation testing is very helpful and cost-effective too and the only time investment in automation testing is only in writing scripts and maintaining them. Thus automation testing needs to be properly planned and scheduled for a successful project.
About the Author: J.B.Rajkumar has more than 15 years of experience in both Academics and Software Testing. He has worked as a Corporate Trainer, Test Lead, QA Manager and QC Manager.
Let us know your comments/suggestions about this article.
=> Visit Here For The Complete Regression Testing Series
Thx mac, but i was trying to download the file but i couldn’t get any proper link, can you plz send me the link to download the file for win xp
Hi
Nice Information.
I am currently working as Lead QA in a Project using Agile Methodology, we also have an onshore-offshore resource model. I just finished a project with Manual Testing and Automated Performance Testing in Aglie Methodology. The Project was successfull however there were so many things to learn and improve which I am trying to implement in my current project. JB it would be great if you could share some automation tools information that you used in your project.
Anyone here has any questions on Agile let me know.
Thanks,
Monit
I agree with the point that in an agile projects, and to support test first, you need the type of testing tools that allow scripting, so that testers can draft the test cases before they even see the feature. This requires a scripting tool/language that is flexible, and maintainable.
We should treat this type of test scripts as we treat production code, it is subject to refactoring, it has bad smells (like code duplication), and it has readability and maintainability, just like the code.
If testers can learn those principles, and can start building a set of commands (vocabulary) associated with domain of the application being built, they can become very productive in writing this type of scripts. It may be slow at the beginning, but this will pay back in the future of the project!
Hi,
I am planning to implement 100% automation on our regression testing phase. The entire architecture is based on C & Unix (scripting) based applications across multiple boxes. Could someone suggest some good automation tools that I can explore.
Hi All,
I am working in a start up company where Testing is started 3months back.I have some queries
1.We are dealing with a number of short term projects,and resources are less in number.
So can we go with any of the open source automation tools to reduce our burden and quicken the testing process
2.What are the best open source tools which can be used for functional and performance testing of Website,Web Applications,Client Server Applications effectively.
Thanks
Madhuri
Hi,
Can anybody provide me the loadRunner software
will be very thankfull to him.
mail id:manish.patidar88@gmail.com
Regards,
Manish
Hi ,
im new to automation. Planning to implement automation in our project. i wish to go for an open source tools. can anyone of you share your views of using an open source tool for testing a java based project.
Thanks in advance.
Regds,
Swetha (Swethakk07@gmail.com)
Great post! this helps me a lot with the whole automation process from start to end. You guys are amazing. Keep posting. Thanks
Hi All,
Please Answer the given following questions –
What is the use of Method:-Reporter Event & Properties for Reporter Event is
1) Filter
2) Reporter Path
Where we use the Reporter Event Please give an practical Example with QTP Pictured view
if it is possible?
What is the use of Smart Identification & How to use Smart Identification in the QTP
Please give an practical Example?
Where we use the Virtual Object & How to use it in the QTP Please give an practical Example?
What is ImageCheck point & Bitmapcheck point the What is the difference between
ImageCheck point & Bitmapcheck point & How to use it in the QTP Please give an practical Example?
What is Text & TextArea checkpoint the What is the difference between
Text & TextArea checkpoint & How to use it in the QTP Please give an practical Example?
What is the use of Business Component, Scripted Component, Application Area in the QTP
& Quality Center. How to use it in the QTP Please give an practical Example?
What is the use of Function Library in the QTP Please give an practical Example?
What is the Object Repository & Object Repository Manager & what is the difference
between Object Repository & Object Repository Manager in the QTP Please give an practical Example?
What is the Call to Copy Action & Call to Exiting Action in QTP & what is the difference
between Call to Copy Action & Call to Exiting Action in the QTP Please give an practical Example?
What is Virtual Object & why & what is the use of Virtual Object in the QTP
Please give an practical Example?
thanks,
Ajay
App changes every sprint and worst part it changes for all environment most time is spent on modifying scripts rather than testing…
Good Material.
-Sridhar Chakilam
Hi All,
Please Answer the given following questions –
What is the use of Method:-Reporter Event & Properties for Reporter Event is
1) Filter
2) Reporter Path
Where we use the Reporter Event Please give an practical Example with QTP Pictured view
if it is possible?
What is the use of Smart Identification & How to use Smart Identification in the QTP
Please give an practical Example?
Where we use the Virtual Object & How to use it in the QTP Please give an practical Example?
What is ImageCheck point & Bitmapcheck point the What is the difference between
ImageCheck point & Bitmapcheck point & How to use it in the QTP Please give an practical Example?
What is Text & TextArea checkpoint the What is the difference between
Text & TextArea checkpoint & How to use it in the QTP Please give an practical Example?
What is the use of Business Component, Scripted Component, Application Area in the QTP
& Quality Center. How to use it in the QTP Please give an practical Example?
What is the use of Function Library in the QTP Please give an practical Example?
What is the Object Repository & Object Repository Manager & what is the difference
between Object Repository & Object Repository Manager in the QTP Please give an practical Example?
What is the Call to Copy Action & Call to Exiting Action in QTP & what is the difference
between Call to Copy Action & Call to Exiting Action in the QTP Please give an practical Example?
What is Virtual Object & why & what is the use of Virtual Object in the QTP
Please give an practical Example?
thanks,
Ajay
any one could please send me installation guide of bugzilla
my mailid:raviaruva@gmail.com
nice post and thx for sharing
highly impressed by prioritied requirements
please provide the documents related to ‘Selenium’ tool.
HI
I am going to work with Agile testing, I am an testing team lead, what type of steps I need to taken for implementing agile methodology into testing, Is any body having the docs or informaiton please share with me
Hi,
I could associate with all these challanges, becuase i have recently worked on one Agile project.
Businsess Analyst kept on changing the requirement and we were given the task to finish the job in 2 weeks, end of the day it become too messy to work.
Why Agile? from business point:
Testing resources are more idle then developers, by this they have productivity improved.
From Architect Point:
Efficient and exhaustive testing can be done.
From Tester point:
i mention only negatives here:
More challanges, after a while very mundane work, no growth in terms of trainings or learnings.
HI….
Really a good post….. All the concepts and the way you had explained are so good and clear. As working for http://www.macrotesting.com as a tester i got so many ideas each and everyday and now your Article is also making my mind to keep on digging in this key point.
You can try Jmeter open source tool for performance testing
In my current organization, (we say) we follow Agile methodology but there are so many holes into it that I wonder whether we really follow it! This article really answered some of the questions in my mind which usually kept coming. Thank you!
GOOD
Excellent material
J.B., I agree with all these challenges. This is why we have created a tool to automate challenge N3. As a seasoned software quality professional I would greatly appreciate your feedback in regards to our tool. TestRigor eliminates the need to even setup regression cycles. If you or anyone reading this thread are open to test driving the tool, please contact me at artem+test6@testrigor.com
This material is very useful of testing professionals.
Thanks
Regards,
Mehul Fozdar
QC Analyst
Why can’t we automate Sprint 1’s work in Sprint 2?
THis way we deal with the ever changing requirements of SPrint 1 atleast.
Has anyone tried this approach?
which Automation tool according to you is best to use during agile phase..
Interesting read. I think a lot of the problems you raise with Agile methodology and automated testing will be solved by our new testing tool, Janova. At http://www.janova.us you will see a new automated testing tool available for free trial, that is founded on agile methodology. At Janova, we pride ourselves on Agile teachings and believe that Janova is the first truly Agile, automated testing tool.
You can make changes to requirements and development on the fly in Janova, and tests automatically update. The business requirements are now the tests themselves, so there are no external file keeping and manually updating of tests. When you change the business requirement, the test changes with it.
Janova just released and is only going to improve with time. We’d love input on how to help it work more in an agile-based environment.
Excellent and easy understantable material
Thanks
MIC,
Bangalore
My life become easy. Only read the material. After reading the material we go for tools……..
“LIFE IS BEAUTIFUL ONLY FOR THOSE WHO KNOW HOW TO CELEBRATE PAIN.”
I completely agree with Amit. In agile, change is happening continously and there is no end to it. We as testers are targeted for this. I do know that Continous Integration supports this continous change with high degree of test automation(Quite challenging to achieve this high degree of test automation).
But what about the tests that need to be carried out in Release phase(Challenge 7) if the product does not freeze on the requirements?
How will the system tester ever get to know when they should start testing with the actual Releases?. One more challenge here is the developers and test automation team get enough time to change their code as part of their sprint planning but the system testers are stressed to release the software within a time frame.
If there are any links which depicts various phase of testing in Agile environment and also the test scope in each phase is highly recommendable to be given as a reply to this query.
Honest input – Thanks.
Another area of concern is project management, but this is outside of the testing role.
Hi,
Could anyone give me links which could describe Agile Testing in detail
Thanks
Very nice post JB. Thanks
I am working on Automation regression testing for over 2 years now, Most of the releases have new changes compared to the previous ones, so we always land up modifying the scripts for every releases. what could be the best solution to avoid these frequent changes of scripts.
Nice inforamation.
Dear Ms.Priya (#5 and #7),
For performance testing, You can go for Badboy! tool.
Do ad-hoc testing (don’t get scared, just want to say do some research (play with the tool) and analysis) on the tool for couple of days. Then from the third day, You’ll find by yourself, How to make best use of the tool and its features that are supported.
Tool supports upto 1000 back ground threads.
But I recommend you to go for 999 (max) background threads. If its really necessary for you to test beyong 999 background threads, Then open (run) the tool again for the second time (means that now you are using two Badboy! tools) and again you can go for one more 999 background threads in max.
Its a free ware.
Definitely it fits your requirement. Because I succeeded in this and overcame from most my repeatition tasks for each and every off-fix releases and official releases.
Let you work on tool for a week or couple of weeks, Then the tool will start working on behalf of you forever.
Regards,
Govardhan Reddy M,
Software Test Engineer,
“Results matter, Efforts won’t”
@muzamil
To start with Agile Testing, you can refer the following link from R.S.Pressman Associates
http://www.rspa.com/spi/index.html
For Latest updates
http://www.agilejournal.com
http://www.agilealliance.org
nice article, i have learnt so many good things from this aricle.
Can anybody please let me know good open source tools for performance testing
Thx in adv
I like the statement from first automation challenge about requirement
“Customers/software folks take liberty to make too many changes to the requirements.”
The biggest challenge I think is in the requirement phase itself.
When you start automation with agile development it’s very difficult to match with the changes.
Hi, all
Can any one guide me a good Test Automation tool to be used for agil Testing. (Iteration based)
(Paid or free)
Actually our system involves lot of fuctionality testing of web application where full regression testing required in each iteration.
Also few requirement changes in each iteration of development.
Thanks
Darshit
what are the goals of test automation??
I am working in Agile Automation Regression Testing nearly 1.4 years with applications build on Flex Technology.
We hardly try to design the scripts within 10 days when we run on the 11th day the script couldn’t find to be utilized (to execute) effeciently as the build get change drastically, where we leave out effort only in the modification of script. In the mean time the manual team who works on the particular feature does the testing and the scripts which is created goes in Dustbin
So tell me whether atleast this Sprint Automation Testing atleast very much effectively been used by the automation scripts we have in hands for the new build.
Selenium is best open source tool for Automation Testing.
OpenSTA is also a good tool for Performance testing.