In this article Author J.B. Rajkumar is sharing his experience in Implementing Automated Regression Testing in the Agile Environment.
Abstract
Recently, when I wanted to start my new Automated Testing Project with four resources, I thought of applying any one of the Agile methodologies. But I was not able to continue because, a series of questions were raised inside my mind. The questions are like “Is it possible to use Agile methodologies in Automated Testing?”, “Can I use traditional tools”, “Should I have to go for open-source tools”, “What are the challenges I have to face if I am implementing automation in Agile Environment”. In this article let us analyze some of challenges we face while implementing Automation with Agile methodologies. Automated testing in the Agile environment stands a risk of becoming chaotic, unstructured and uncontrolled.
Agile Projects present their own challenges to the Automation team; Unclear project scope, Multiple iterations, Minimal documentation, early and frequent Automation needs and active stakeholder involvement all demand lot of challenges from the Automation Team. Some of these challenges are:
Challenge 1: Requirement Phase
Test Automation developer captures requirements in the form of “user stories”, which are brief descriptions of customer-relevant functionality.
Each requirement has to be prioritized as follows:
High: These are mission critical requirements that absolutely have to be done in the first release
Medium: These are requirements that are important but can be worked around until implemented.
Low: These are requirements that are nice-to-have but not critical to the operation of the software.
Once priories are established, the release “iterations” are planned. Normally, each Agile release iteration takes between 1 to 3 months to deliver. Customers/software folks take liberty to make too many changes to the requirements. Sometimes, these changes are so volatile that the iterations are bumped off. These changes are greater challenges in implementing Agile Automation testing process.
Challenge 2: Selecting the Right Tools
Traditional, test-last tools with record-and-playback-features force teams to wait until after the software is done. More over, traditional test automation tools don’t work for an Agile context because they solve traditional problems, and those are different from the challenges facing Agile Automation teams. Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. So the choice of testing tools becomes critical for reaping the efficiency and quality benefits of agile.
Challenge 3: Script Development Phase
The Automation testers, developers, business analysts and project stakeholders all contribute to kick-off meetings where “user-stories” are selected to next sprint. Once the “user-stories” are selected for the sprint, they are used as the basis for a set of tests.
As functionality grows with each iteration, regression testing must be performed to ensure that existing functionality has not been impacted by the introduction of new functionality in each iteration cycle. The scale of the regression testing grows with each sprint and ensures that this remains a manageable task the test team use the test automation for the regression suite.
Challenge 4: Resource Management
The Agile approach requires a mixture of testing skills, that is, test resource will be required to define unclear scenarios and test cases, conduct manual testing alongside developers, write automated regression tests and execute the automated regression packages. As the project progresses, specialist skills will also be required to cover further test areas that might include integration and performance testing. There should be an appropriate mix of domain specialist who plan and gather requirements. The challenging part in the Resource management is to find out test resources with multiple skills and
allocate them.
Challenge 5: Communication
Good communication must exist among Automation testing team, developers, business analysts and stake holders. There must be highly collaborative interaction between client and the delivery teams. More client involvement implies more suggestions or changes from the client. It implies more bandwidth for communication. The key challenge is that the process should be able to capture and effectively implement all the changes and data integrity needs to be retained. In traditional testing, developers and testers are like oil and water, but in agile environment, the challenging task is that they both must work together to achieve the target.
Challenge 6: Daily Scrum Meeting
Daily Scrum Meeting is one of the key activities in Agile Process. Teams do meet for 15 minutes stand up sessions. What is the effectiveness of these meetings? How far these meetings help Automation practice Developers?
Challenge 7: Release Phase
The aim of Agile project is to deliver a basic working product as quickly as possible and then to go through a process of continual improvement. This means that there is no single release phase for a product. The challenging part lies in integration testing and acceptance testing of the product.
If we can meet these challenges in a well optimized manner, then Automated Regression Testing in Agile environment is an excellent opportunity for QA to take leadership of the agile processes. It is better placed to bridge the gap between users and developers, understand both what is required, how it can be achieved and how it can be assured prior to deployment. Automation practice should have a vested interest in both the how and the result, as well as continuing to assure that the whole evolving system meets business objectives and is fit for purpose.
About the Author:
J.B.Rajkumar has more than 15 years of experience in both Academics and Software Testing. He has worked as Corporate Trainer, Test Lead, QA Manager and QC Manager. He is a frequent speaker for International Conferences, Colleges, Universities and Software Industries. Presently he is with Automation Practice, in one of the top MNCs.
Related Posts:
33 comments ↓
Nice inforamation.
I like the statement from first automation challenge about requirement
“Customers/software folks take liberty to make too many changes to the requirements.”
The biggest challenge I think is in the requirement phase itself.
When you start automation with agile development it’s very difficult to match with the changes.
I agree with the point that in an agile projects, and to support test first, you need the type of testing tools that allow scripting, so that testers can draft the test cases before they even see the feature. This requires a scripting tool/language that is flexible, and maintainable.
We should treat this type of test scripts as we treat production code, it is subject to refactoring, it has bad smells (like code duplication), and it has readability and maintainability, just like the code.
If testers can learn those principles, and can start building a set of commands (vocabulary) associated with domain of the application being built, they can become very productive in writing this type of scripts. It may be slow at the beginning, but this will pay back in the future of the project!
In my current organization, (we say) we follow Agile methodology but there are so many holes into it that I wonder whether we really follow it! This article really answered some of the questions in my mind which usually kept coming. Thank you!
Can anybody please let me know good open source tools for performance testing
Thx in adv
You can try Jmeter open source tool for performance testing
Thx mac, but i was trying to download the file but i couldn’t get any proper link, can you plz send me the link to download the file for win xp
Hi,
Could anyone give me links which could describe Agile Testing in detail
Thanks
@muzamil
To start with Agile Testing, you can refer the following link from R.S.Pressman Associates
http://www.rspa.com/spi/index.html
For Latest updates
http://www.agilejournal.com
http://www.agilealliance.org
This material is very useful of testing professionals.
Thanks
Regards,
Mehul Fozdar
QC Analyst
Good Material.
-Sridhar Chakilam
Excellent material
Excellent and easy understantable material
Thanks
MIC,
Bangalore
HI….
Really a good post….. All the concepts and the way you had explained are so good and clear. As working for http://www.macrotesting.com as a tester i got so many ideas each and everyday and now your Article is also making my mind to keep on digging in this key point.
OpenSTA is also a good tool for Performance testing.
nice post and thx for sharing
highly impressed by prioritied requirements
Hi
Nice Information.
I am currently working as Lead QA in a Project using Agile Methodology, we also have an onshore-offshore resource model. I just finished a project with Manual Testing and Automated Performance Testing in Aglie Methodology. The Project was successfull however there were so many things to learn and improve which I am trying to implement in my current project. JB it would be great if you could share some automation tools information that you used in your project.
Anyone here has any questions on Agile let me know.
Thanks,
Monit
what are the goals of test automation??
any one could please send me installation guide of bugzilla
my mailid:raviaruva@gmail.com
Hi,
I could associate with all these challanges, becuase i have recently worked on one Agile project.
Businsess Analyst kept on changing the requirement and we were given the task to finish the job in 2 weeks, end of the day it become too messy to work.
Why Agile? from business point:
Testing resources are more idle then developers, by this they have productivity improved.
From Architect Point:
Efficient and exhaustive testing can be done.
From Tester point:
i mention only negatives here:
More challanges, after a while very mundane work, no growth in terms of trainings or learnings.
I completely agree with Amit. In agile, change is happening continously and there is no end to it. We as testers are targeted for this. I do know that Continous Integration supports this continous change with high degree of test automation(Quite challenging to achieve this high degree of test automation).
But what about the tests that need to be carried out in Release phase(Challenge 7) if the product does not freeze on the requirements?
How will the system tester ever get to know when they should start testing with the actual Releases?. One more challenge here is the developers and test automation team get enough time to change their code as part of their sprint planning but the system testers are stressed to release the software within a time frame.
If there are any links which depicts various phase of testing in Agile environment and also the test scope in each phase is highly recommendable to be given as a reply to this query.
Dear Ms.Priya (#5 and #7),
For performance testing, You can go for Badboy! tool.
Do ad-hoc testing (don’t get scared, just want to say do some research (play with the tool) and analysis) on the tool for couple of days. Then from the third day, You’ll find by yourself, How to make best use of the tool and its features that are supported.
Tool supports upto 1000 back ground threads.
But I recommend you to go for 999 (max) background threads. If its really necessary for you to test beyong 999 background threads, Then open (run) the tool again for the second time (means that now you are using two Badboy! tools) and again you can go for one more 999 background threads in max.
Its a free ware.
Definitely it fits your requirement. Because I succeeded in this and overcame from most my repeatition tasks for each and every off-fix releases and official releases.
Let you work on tool for a week or couple of weeks, Then the tool will start working on behalf of you forever.
Regards,
Govardhan Reddy M,
Software Test Engineer,
“Results matter, Efforts won’t”
Hi All,
Please Answer the given following questions -
What is the use of Method:-Reporter Event & Properties for Reporter Event is
1) Filter
2) Reporter Path
Where we use the Reporter Event Please give an practical Example with QTP Pictured view
if it is possible?
What is the use of Smart Identification & How to use Smart Identification in the QTP
Please give an practical Example?
Where we use the Virtual Object & How to use it in the QTP Please give an practical Example?
What is ImageCheck point & Bitmapcheck point the What is the difference between
ImageCheck point & Bitmapcheck point & How to use it in the QTP Please give an practical Example?
What is Text & TextArea checkpoint the What is the difference between
Text & TextArea checkpoint & How to use it in the QTP Please give an practical Example?
What is the use of Business Component, Scripted Component, Application Area in the QTP
& Quality Center. How to use it in the QTP Please give an practical Example?
What is the use of Function Library in the QTP Please give an practical Example?
What is the Object Repository & Object Repository Manager & what is the difference
between Object Repository & Object Repository Manager in the QTP Please give an practical Example?
What is the Call to Copy Action & Call to Exiting Action in QTP & what is the difference
between Call to Copy Action & Call to Exiting Action in the QTP Please give an practical Example?
What is Virtual Object & why & what is the use of Virtual Object in the QTP
Please give an practical Example?
thanks,
Ajay
Hi All,
Please Answer the given following questions -
What is the use of Method:-Reporter Event & Properties for Reporter Event is
1) Filter
2) Reporter Path
Where we use the Reporter Event Please give an practical Example with QTP Pictured view
if it is possible?
What is the use of Smart Identification & How to use Smart Identification in the QTP
Please give an practical Example?
Where we use the Virtual Object & How to use it in the QTP Please give an practical Example?
What is ImageCheck point & Bitmapcheck point the What is the difference between
ImageCheck point & Bitmapcheck point & How to use it in the QTP Please give an practical Example?
What is Text & TextArea checkpoint the What is the difference between
Text & TextArea checkpoint & How to use it in the QTP Please give an practical Example?
What is the use of Business Component, Scripted Component, Application Area in the QTP
& Quality Center. How to use it in the QTP Please give an practical Example?
What is the use of Function Library in the QTP Please give an practical Example?
What is the Object Repository & Object Repository Manager & what is the difference
between Object Repository & Object Repository Manager in the QTP Please give an practical Example?
What is the Call to Copy Action & Call to Exiting Action in QTP & what is the difference
between Call to Copy Action & Call to Exiting Action in the QTP Please give an practical Example?
What is Virtual Object & why & what is the use of Virtual Object in the QTP
Please give an practical Example?
thanks,
Ajay
Hi, all
Can any one guide me a good Test Automation tool to be used for agil Testing. (Iteration based)
(Paid or free)
Actually our system involves lot of fuctionality testing of web application where full regression testing required in each iteration.
Also few requirement changes in each iteration of development.
Thanks
Darshit
nice article, i have learnt so many good things from this aricle.
Hi All,
I am working in a start up company where Testing is started 3months back.I have some queries
1.We are dealing with a number of short term projects,and resources are less in number.
So can we go with any of the open source automation tools to reduce our burden and quicken the testing process
2.What are the best open source tools which can be used for functional and performance testing of Website,Web Applications,Client Server Applications effectively.
Thanks
Madhuri
HI
I am going to work with Agile testing, I am an testing team lead, what type of steps I need to taken for implementing agile methodology into testing, Is any body having the docs or informaiton please share with me
GOOD
Interesting read. I think a lot of the problems you raise with Agile methodology and automated testing will be solved by our new testing tool, Janova. At http://www.janova.us you will see a new automated testing tool available for free trial, that is founded on agile methodology. At Janova, we pride ourselves on Agile teachings and believe that Janova is the first truly Agile, automated testing tool.
You can make changes to requirements and development on the fly in Janova, and tests automatically update. The business requirements are now the tests themselves, so there are no external file keeping and manually updating of tests. When you change the business requirement, the test changes with it.
Janova just released and is only going to improve with time. We’d love input on how to help it work more in an agile-based environment.
Honest input – Thanks.
Another area of concern is project management, but this is outside of the testing role.
I am working in Agile Automation Regression Testing nearly 1.4 years with applications build on Flex Technology.
We hardly try to design the scripts within 10 days when we run on the 11th day the script couldn’t find to be utilized (to execute) effeciently as the build get change drastically, where we leave out effort only in the modification of script. In the mean time the manual team who works on the particular feature does the testing and the scripts which is created goes in Dustbin
So tell me whether atleast this Sprint Automation Testing atleast very much effectively been used by the automation scripts we have in hands for the new build.
App changes every sprint and worst part it changes for all environment most time is spent on modifying scripts rather than testing…
Leave a Comment