Automated Regression Testing Challenges in Agile Environment

In this article, I (J.B.Rajkumar) have shared my experience about Implementing Automated Regression Testing in the Agile Environment. 

Abstract
Recently, when I wanted to start my new Automated Testing Project with four resources, I thought of applying any one of the Agile methodologies. But I was not able to continue because a series of questions were raised in my mind. 

The questions are like “Is it possible to use Agile methodologies in Automated Testing?”, “Can I use traditional tools”, “Should I have to go for open-source tools”, “What are the challenges I have to face if I am implementing automation in the Agile Environment”. In this article let us analyze some of the challenges we face while implementing Automation with Agile methodologies. Automated testing in the Agile environment stands as a risk of becoming chaotic, unstructured and uncontrolled.

Challenges In Agile Environment

Agile

Agile Projects present their own challenges to the Automation team. Agile methodology emphasizes on team collaboration and frequent delivery of a product. Factors like unclear project scope, Multiple iterations, Minimal documentation, early and frequent Automation needs and active stakeholder involvement, etc., demand a lot of challenges from the Automation Team.

Testing Challenges in Agile Environment

There are several Agile challenges to the Automation team. However, few of them are briefed below.

Challenge 1: Requirement Phase

Test Automation developer captures the requirements in the form of “user stories”, which are brief descriptions of the customer-relevant functionality.

Each requirement has to be prioritized as follows:

High: These are mission-critical requirements that absolutely needs to be done in the first release
Medium: These are the requirements that are important, but can be worked around until implemented.
Low: These are the requirements that are nice-to-have but not critical to the operation of the software.

Once priories are established, the release “iterations” are planned. Normally, each Agile release iteration takes between 1 to 3 months to deliver. Customer/software folks take the liberty to make too many changes to the requirements. Sometimes, these changes are so volatile, that the iterations are bumped off. These changes are greater challenges in implementing Agile Automation testing process.

Challenge 2: Selecting the Right Tools

Traditional, test-last tools with record-and-playback-features force teams to wait until after the software is done. Moreover, traditional test automation tools don’t work for an Agile context because they solve traditional problems which are different from the challenges facing Agile Automation teams.

Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. So the choice of testing tools becomes critical for reaping the efficiency and quality benefits of agile.


Challenge 3: Script Development Phase

Automation testers, developers, business analysts and project stakeholders altogether contribute to kick-off meetings where “user-stories” are selected to the next sprint. Once the “user-stories” are selected for the sprint, they are used as the basis for a set of tests.

As the functionality grows with each iteration, regression testing must be performed to ensure that existing functionality has not been impacted by the introduction of new functionality in each iteration cycle. The scale of the regression testing grows with each sprint and ensures that this remains as a manageable task and the test team uses the test automation for the regression suite.

Challenge 4: Resource Management

The Agile approach requires a mixture of testing skills, that is, test resource will be required to define unclear scenarios and test cases, conduct manual testing alongside developers, write automated regression tests and execute the automated regression packages. As the project progresses, specialist skills will also be required to cover further test areas that might include integration and performance testing.

There should be an appropriate mix of domain specialists who plan and gather the requirements. The challenging part in the Resource management is to find out test resources with multiple skills and allocate them.

Challenge 5: Communication

Good communication must exist among Automation testing team, developers, business analysts and stakeholders. There must be the highly collaborative interaction between client and the delivery teams. More client involvement implies more suggestions or changes from the client.  And it implies more bandwidth for communication.

The key challenge is that the process should be able to capture and effectively implement all the changes and the data integrity needs to be retained. In traditional testing, developers and testers are like oil and water, but in an agile environment, the challenging task is that they both must work together to achieve the target.

Challenge 6: Daily Scrum Meeting

Daily Scrum Meeting is one of the key activities in the Agile Process. Teams do meet for 15 minutes stand up sessions. What is the effectiveness of these meetings? How far these meetings help Automation practice Developers? etc., are discussed in this meeting.

Challenge 7: Release Phase

The aim of the Agile project is to deliver a basic working product as quickly as possible and then to go through a process of continual improvement. This means that there is no single release phase for a product. The challenging part lies in integration testing and acceptance testing of the product.

Conclusion

If we can meet these challenges in a well-optimized manner, then Automated Regression Testing in the Agile environment is an excellent opportunity for the QA to take up leadership of the agile processes. It is better placed to bridge the gap between the users and developers, understand what is required, how it can be achieved and how it can be assured prior to deployment.

Automation practice should have a vested interest in both, the result, as well as continuing to assure that the whole evolving system meets business objectives and is fit for the purpose.

About the Author:
J.B.Rajkumar has more than 15 years of experience in both Academics and Software Testing. He has worked as a Corporate Trainer, Test Lead, QA Manager and QC Manager. He is a frequent speaker for International Conferences, Colleges, Universities and Software Industries. Presently he is with Automation Practice, in one of the top MNCs.

Let us know your comments/suggestions about this article.




Recommended reading

42 comments ↓

#1 Arman

Nice inforamation.

#2 Shrikant

I like the statement from first automation challenge about requirement

“Customers/software folks take liberty to make too many changes to the requirements.”

The biggest challenge I think is in the requirement phase itself.
When you start automation with agile development it’s very difficult to match with the changes.

#3 Khaldoon Aqel

I agree with the point that in an agile projects, and to support test first, you need the type of testing tools that allow scripting, so that testers can draft the test cases before they even see the feature. This requires a scripting tool/language that is flexible, and maintainable.

We should treat this type of test scripts as we treat production code, it is subject to refactoring, it has bad smells (like code duplication), and it has readability and maintainability, just like the code.

If testers can learn those principles, and can start building a set of commands (vocabulary) associated with domain of the application being built, they can become very productive in writing this type of scripts. It may be slow at the beginning, but this will pay back in the future of the project!

#4 Ambarish Karnik

In my current organization, (we say) we follow Agile methodology but there are so many holes into it that I wonder whether we really follow it! This article really answered some of the questions in my mind which usually kept coming. Thank you!

#5 Priya

Can anybody please let me know good open source tools for performance testing

Thx in adv

#6 mac

You can try Jmeter open source tool for performance testing

#7 Priya

Thx mac, but i was trying to download the file but i couldn’t get any proper link, can you plz send me the link to download the file for win xp

#8 muzamil

Hi,
Could anyone give me links which could describe Agile Testing in detail

Thanks

#9 J.B.Rajkumar

@muzamil

To start with Agile Testing, you can refer the following link from R.S.Pressman Associates

http://www.rspa.com/spi/index.html

For Latest updates

http://www.agilejournal.com
http://www.agilealliance.org

#10 Mehul Fozdar

This material is very useful of testing professionals.

Thanks
Regards,
Mehul Fozdar
QC Analyst

#11 sridhar chakilam

Good Material.

-Sridhar Chakilam

#12 qtpconsultant@gmail.com

Excellent material

#13 qtpconsultant@gmail.com

Excellent and easy understantable material

Thanks
MIC,
Bangalore

#14 sakthi

HI….
Really a good post….. All the concepts and the way you had explained are so good and clear. As working for http://www.macrotesting.com as a tester i got so many ideas each and everyday and now your Article is also making my mind to keep on digging in this key point.

#15 VijayD

OpenSTA is also a good tool for Performance testing.

#16 imutou

nice post and thx for sharing
highly impressed by prioritied requirements

#17 Monit Sao

Hi

Nice Information.
I am currently working as Lead QA in a Project using Agile Methodology, we also have an onshore-offshore resource model. I just finished a project with Manual Testing and Automated Performance Testing in Aglie Methodology. The Project was successfull however there were so many things to learn and improve which I am trying to implement in my current project. JB it would be great if you could share some automation tools information that you used in your project.
Anyone here has any questions on Agile let me know.

Thanks,
Monit

#18 sam_mac

what are the goals of test automation??

#19 raviaruva

any one could please send me installation guide of bugzilla

my mailid:raviaruva@gmail.com

#20 Amit Mishra

Hi,
I could associate with all these challanges, becuase i have recently worked on one Agile project.
Businsess Analyst kept on changing the requirement and we were given the task to finish the job in 2 weeks, end of the day it become too messy to work.

Why Agile? from business point:
Testing resources are more idle then developers, by this they have productivity improved.
From Architect Point:
Efficient and exhaustive testing can be done.
From Tester point:
i mention only negatives here:
More challanges, after a while very mundane work, no growth in terms of trainings or learnings.

#21 Shanthala

I completely agree with Amit. In agile, change is happening continously and there is no end to it. We as testers are targeted for this. I do know that Continous Integration supports this continous change with high degree of test automation(Quite challenging to achieve this high degree of test automation).
But what about the tests that need to be carried out in Release phase(Challenge 7) if the product does not freeze on the requirements?
How will the system tester ever get to know when they should start testing with the actual Releases?. One more challenge here is the developers and test automation team get enough time to change their code as part of their sprint planning but the system testers are stressed to release the software within a time frame.
If there are any links which depicts various phase of testing in Agile environment and also the test scope in each phase is highly recommendable to be given as a reply to this query.

#22 Govardhan Reddy M

Dear Ms.Priya (#5 and #7),
For performance testing, You can go for Badboy! tool.
Do ad-hoc testing (don’t get scared, just want to say do some research (play with the tool) and analysis) on the tool for couple of days. Then from the third day, You’ll find by yourself, How to make best use of the tool and its features that are supported.

Tool supports upto 1000 back ground threads.
But I recommend you to go for 999 (max) background threads. If its really necessary for you to test beyong 999 background threads, Then open (run) the tool again for the second time (means that now you are using two Badboy! tools) and again you can go for one more 999 background threads in max.

Its a free ware.
Definitely it fits your requirement. Because I succeeded in this and overcame from most my repeatition tasks for each and every off-fix releases and official releases.
Let you work on tool for a week or couple of weeks, Then the tool will start working on behalf of you forever.

Regards,
Govardhan Reddy M,
Software Test Engineer,
“Results matter, Efforts won’t”

#23 Ajay

Hi All,
Please Answer the given following questions –

What is the use of Method:-Reporter Event & Properties for Reporter Event is

1) Filter

2) Reporter Path

Where we use the Reporter Event Please give an practical Example with QTP Pictured view
if it is possible?

What is the use of Smart Identification & How to use Smart Identification in the QTP
Please give an practical Example?

Where we use the Virtual Object & How to use it in the QTP Please give an practical Example?

What is ImageCheck point & Bitmapcheck point the What is the difference between
ImageCheck point & Bitmapcheck point & How to use it in the QTP Please give an practical Example?

What is Text & TextArea checkpoint the What is the difference between
Text & TextArea checkpoint & How to use it in the QTP Please give an practical Example?

What is the use of Business Component, Scripted Component, Application Area in the QTP
& Quality Center. How to use it in the QTP Please give an practical Example?

What is the use of Function Library in the QTP Please give an practical Example?

What is the Object Repository & Object Repository Manager & what is the difference
between Object Repository & Object Repository Manager in the QTP Please give an practical Example?

What is the Call to Copy Action & Call to Exiting Action in QTP & what is the difference
between Call to Copy Action & Call to Exiting Action in the QTP Please give an practical Example?

What is Virtual Object & why & what is the use of Virtual Object in the QTP
Please give an practical Example?

thanks,
Ajay

#24 Ajay

Hi All,
Please Answer the given following questions –

What is the use of Method:-Reporter Event & Properties for Reporter Event is

1) Filter

2) Reporter Path

Where we use the Reporter Event Please give an practical Example with QTP Pictured view
if it is possible?

What is the use of Smart Identification & How to use Smart Identification in the QTP
Please give an practical Example?

Where we use the Virtual Object & How to use it in the QTP Please give an practical Example?

What is ImageCheck point & Bitmapcheck point the What is the difference between
ImageCheck point & Bitmapcheck point & How to use it in the QTP Please give an practical Example?

What is Text & TextArea checkpoint the What is the difference between
Text & TextArea checkpoint & How to use it in the QTP Please give an practical Example?

What is the use of Business Component, Scripted Component, Application Area in the QTP
& Quality Center. How to use it in the QTP Please give an practical Example?

What is the use of Function Library in the QTP Please give an practical Example?

What is the Object Repository & Object Repository Manager & what is the difference
between Object Repository & Object Repository Manager in the QTP Please give an practical Example?

What is the Call to Copy Action & Call to Exiting Action in QTP & what is the difference
between Call to Copy Action & Call to Exiting Action in the QTP Please give an practical Example?

What is Virtual Object & why & what is the use of Virtual Object in the QTP
Please give an practical Example?

thanks,
Ajay

#25 Darshit

Hi, all

Can any one guide me a good Test Automation tool to be used for agil Testing. (Iteration based)
(Paid or free)
Actually our system involves lot of fuctionality testing of web application where full regression testing required in each iteration.
Also few requirement changes in each iteration of development.

Thanks
Darshit

#26 geek blogger

nice article, i have learnt so many good things from this aricle.

#27 Madhuri

Hi All,
I am working in a start up company where Testing is started 3months back.I have some queries
1.We are dealing with a number of short term projects,and resources are less in number.
So can we go with any of the open source automation tools to reduce our burden and quicken the testing process
2.What are the best open source tools which can be used for functional and performance testing of Website,Web Applications,Client Server Applications effectively.

Thanks
Madhuri

#28 Rao BT

HI
I am going to work with Agile testing, I am an testing team lead, what type of steps I need to taken for implementing agile methodology into testing, Is any body having the docs or informaiton please share with me

#29 shoba

GOOD

#30 Blake Lusenhop

Interesting read. I think a lot of the problems you raise with Agile methodology and automated testing will be solved by our new testing tool, Janova. At http://www.janova.us you will see a new automated testing tool available for free trial, that is founded on agile methodology. At Janova, we pride ourselves on Agile teachings and believe that Janova is the first truly Agile, automated testing tool.

You can make changes to requirements and development on the fly in Janova, and tests automatically update. The business requirements are now the tests themselves, so there are no external file keeping and manually updating of tests. When you change the business requirement, the test changes with it.

Janova just released and is only going to improve with time. We’d love input on how to help it work more in an agile-based environment.

#31 Cam

Honest input – Thanks.

Another area of concern is project management, but this is outside of the testing role.

#32 Radhika

I am working in Agile Automation Regression Testing nearly 1.4 years with applications build on Flex Technology.

We hardly try to design the scripts within 10 days when we run on the 11th day the script couldn’t find to be utilized (to execute) effeciently as the build get change drastically, where we leave out effort only in the modification of script. In the mean time the manual team who works on the particular feature does the testing and the scripts which is created goes in Dustbin

So tell me whether atleast this Sprint Automation Testing atleast very much effectively been used by the automation scripts we have in hands for the new build.

#33 Abhi

App changes every sprint and worst part it changes for all environment most time is spent on modifying scripts rather than testing…

#34 bala

Very nice post JB. Thanks
I am working on Automation regression testing for over 2 years now, Most of the releases have new changes compared to the previous ones, so we always land up modifying the scripts for every releases. what could be the best solution to avoid these frequent changes of scripts.

#35 AR BALOCH

My life become easy. Only read the material. After reading the material we go for tools……..

“LIFE IS BEAUTIFUL ONLY FOR THOSE WHO KNOW HOW TO CELEBRATE PAIN.”

#36 Shwetha

Hi ,

im new to automation. Planning to implement automation in our project. i wish to go for an open source tools. can anyone of you share your views of using an open source tool for testing a java based project.

Thanks in advance.

Regds,
Swetha (Swethakk07@gmail.com)

#37 Manish Patidar

Hi,
Can anybody provide me the loadRunner software
will be very thankfull to him.

mail id:manish.patidar88@gmail.com

Regards,
Manish

#38 Pradeep

which Automation tool according to you is best to use during agile phase..

#39 Pradeep

please provide the documents related to ‘Selenium’ tool.

#40 Kranthi Kumar

Selenium is best open source tool for Automation Testing.

#41 Arvind

Hi,
I am planning to implement 100% automation on our regression testing phase. The entire architecture is based on C & Unix (scripting) based applications across multiple boxes. Could someone suggest some good automation tools that I can explore.

#42 Artem

J.B., I agree with all these challenges. This is why we have created a tool to automate challenge N3. As a seasoned software quality professional I would greatly appreciate your feedback in regards to our tool. TestRigor eliminates the need to even setup regression cycles. If you or anyone reading this thread are open to test driving the tool, please contact me at artem+test6@testrigor.com

Leave a Comment