Now is the era of Test Automation. Most of the testing projects are trying to translate their manual test cases to automated ones to improve productivity and coverage.
One of the key steps to commence Automation Testing is – selecting the appropriate test cases and determining the ROI (Return On Investment).
What to expect from this article?
In this article, I have tried to cite down few important points based on my experience to help you select the correct candidate for automation and determine various other factors that will yield better test results and benefits.
Why Automated Testing?
Automation does not overpower or replaces Manual Testing but it compliments it. Like Manual, Automation needs a strategy with proper planning, monitoring & control. Automation, when implemented correctly, can become an asset to the team, project and ultimately to the organization.
There are many advantages of Automation; here are few important to mention:
- Useful to execute routine tasks like Smoke tests and Regression tests.
- Useful in preparing the Test data.
- Helps to execute the Test cases which involve complex business logic.
- Good to execute the cross-platform test cases (like different OS, browsers, etc.)
- Great to execute the test cases which are a bit difficult to execute manually.
- When the number of iterations of the test case executions is not known.
Many a time stakeholders feel that test automation acts as a support tool for Manual Testing, so it’s vital to understand that automation is the best way to increase the effectiveness, efficiency, and coverage of testing. It not only saves time but also improves accuracy as repetitive tasks via the manual approach can prone to human errors and can be time-consuming.
Candidates For Automation
Basic Mistake to Avoid:
One of the most basic mistakes which testers make is NOT Selecting the correct test cases for automation.
Don’t just select any test suite. Analyze the test cases thoroughly and select the candidates for automation considering the most important factor i.e. ROI. First, we should understand and find out the ways for getting a higher and positive ROI.
(ROI – Return On Investment – It is a calculation of benefits in terms of cost-saving, increased efficiency, and quality)
There is no standard procedure for determining the correct test cases for automation. It all depends on the application you are testing.
Based on my experience, I have tried to chalk down few steps which may provide some insight for selecting the test cases and ultimately progress to achieve a positive ROI for automation.
See Also = > How to Translate Manual Test Cases into Automation Scripts?
What You Will Learn:
How To Select Correct Test Cases For Automation Testing
Step 1:
Identify the parameters on which you will base your test case as a candidate for automation.
As of now, I am identifying the below parameters, you can have your own parameters depending on your application.
- Test case executed with different sets of data.
- Test case executed with different browsers.
- Test case executed with different environments.
- Test case executed with complex business logic
- Test case executed with a different set of users
- Test case involves a large amount of data
- Test case has any dependency
- Test case requires Special data
Step 2:
Break each application into modules. For each module, analyze and try to identify the test cases which should be automated based on the parameters. This list will vary for projects to projects and can also be enhanced to suit your needs:
Fig 1.0
Y – Yes
N – No
In a similar way, for all modules, this list can be used to identify the automation candidate test cases.
Step 3:
Consolidate and group the number of test cases for each module shown below
Fig 2.0
Figure 2.0 is quite straightforward and self-explanatory. Here I am trying to quantify the details and giving an estimate to finish the testing manually.
Step 4:
Once you have identified all the granular level details, you can present them in the below way. We are now progressing to calculate the ROI.
Fig 3.0:
We should also take into account the below attributes which form the basis of deterring the ROI:
- Purchasing and licensing cost of the tool
- Time to develop the scripts
- Time to maintain the scripts.
- Time to analyze the results manually and automatically
- Time and cost to train the resources.
- Management overheads
Test Automation ROI Calculation Example
In most of the cases, ROI is calculated for 5 years, but it’s not mandatory. Based on the above factors, let me try to elaborate on the calculation of ROI for 5 years. As usual, you can always tailor and enhance it.
*ROI = (Cumulative saving / Investment through automation)*100
Manual To Automation Testing – What Are The Process Challenges?
I have tried to cite the points, which I feel, are a big challenge, when we try to automate a test suite.
#1) Automation Need: Every test team is unique and has an exclusive need for automation. We cannot develop a fixed standard, but we can tailor a standard which will suit our need. Because of this reason, automation does require good support from the management as well as from the development team.
#2) Automating the complete application: Automating a 100% application is a big task. Not that it is impossible, but it requires proper planning and monitoring and of course; some time. There are lots of permutations and combinations of data, n number of environments with n number of authentication and authorization attributes that need to be validated and hence require a strategy to automate.
#3) Manual Vs Automation mentality: “We normally automate which is important and repetitive, but we prefer to test the important functionality manually”. Confused? Even I am!! But this is a fact. We should have criteria which will decide which are the important test cases. These criteria can be based on multiple factors like complex business logic, areas that are of more interest to clients, risk-prone areas, etc.
#4) Deciding on the framework: Designing the framework is the most important facet of automation. I believe we should devote relatively more time to develop the framework than to script. Whenever we develop the automation plan, framework designing should be the main focus.
PLAN to design the framework. Identify and make a checklist of the items which will form the framework. If the framework is rock solid, scripting and maintaining become easy.
#5) Knowledge of the team: Whenever we think of automation, we immediately jump to learn the programming language or scripting language. Learning this language will definitely help but more emphasis should be on building and developing logic.
Automation should not be the responsibility of some handful of resources, rather an entire team should contribute towards it. This will help not only to enhance the skills of the resources but also keep them motivated.
#6) Reporting: Every tool has a standard to report the test results. To customize it; is a challenging task. Reporting the test results also requires coordination and maintenance which adds to the cost.
#7) Trust: We should trust our automation stuff. We invest man-hours to build an automation suite but still, we don’t believe in the test results. Efforts should be given to maintaining the scripts. Also, we should see that the team who is doing the manual testing of the application, should be involved to automate it as they know their application.
Most of the time, a third-team does the automation so the actual testing team is not aware of the scripts and finally end up with running the tests manually because they feel follow up on the scripts, add up to their tasks
See Also => Manual and Automation Testing Challenges.
Conclusion
In most of the cases, we prefer to Automate the Regression suite (here are some challenges in automating regression suite in an agile environment) as it contains a bigger number of test cases. In that case, we can break down the regression suits into smaller suits and decide to run the appropriate suite as per the release requirement.
Suppose a regression suite contains 1500 test cases, you can break it to 3 suits of 500 test cases per suit and automate it.
Instead of automating the entire suite, you can opt for phase-wise automation. In other words, you can follow the prototype model for developing the automation suite. Create a structure or framework with the implementation of fewer numbers of test cases and start using that and gradually enhance it by adding more test cases to it.
We should follow the Deming wheel (PDCA cycle) for automation as well. Being an ongoing activity, emphasis should be laid on building the framework properly which will result in ease of maintenance and implementing new features.
It requires proper support from the dev team and from management as well. We should encourage the testing team to contribute the most to Automation Testing as they know their product more than anyone else.
About the Author: This is a guest article by Shilpa Chatterjee Roy. She is working in the Software Testing field for the past 8.5 years in various domains.
I hope we have simplified this ‘Manual to Automation testing’ process. Please feel free to share your experience and thoughts on how you overcome the process challenges.
The automation ROI example provided is very helpful. It shows how automation is clearly a winner over long run.
thanks for explaining this in so simple words.
How did you calculate the values for “Total automation effort (In hours)” in the ROI table? Thanks!
challenges are identified correctly and also like the solutions provided to those.
great article.
Excellent blog, everything is very systematically that a fresher can also get idea from basic to advance level concept. Great Work, Keep on going.
Really nice work.
Great help. Keep it up , please..
Thanks,
sunil
Really a great help to identify and differentiate between the two testing processes.
Thanks.
Sujata
Thanks for your patience providing detailed information. It also helps those who are facing interviews
Thanks sir,
sir can we use selenium web driver for small application?
suppose my application is like online project management system.
please help me i am confuse
Thanks All for commenting on my article!
@Bibhishan,
Selenium 2 (web driver) is a browser automation tool so you can use it no matter how big your application is. But before putting your efforts into it, its better to identify how frequently you actually run the automation candidate test cases. Sometimes it happens that for smaller applications, when you run manually the tests are executed in lesser time, so management are not so keen to automate those stuffs, but if you could present the benefits of automating that small application, it should not be a problem.
As far as cost is concerned, selenium being an open source tool, should not be a concern.
Impressed by metrics given.Sounds good and practical,Keep posting this kind of articles.It helps testers to gain more knowledge on Metrics.
Really Nice “Thanks for sharing :-)“
Really nice article. Thanks for sharing and helping us to improve the new latest concepts. keep us shared.
A Good and Perfect Article, Every Management who looks for Automation would have read this!! Good Work Roy:)
Keep us posted a very good information like this!!!
Really it’s good blog…. thank u sir …..
I need a Best testcase template. Can u please share the same to my mail. rahulrakesh16@gmail.com
Thanks in advance.
Rakesh S M
Thank mam,
For this great information…
we always looking for this kind of great information…
It’s really good article. It gives clear idea about selection of test cases for Automation script and also ROI
Excellent article. This is what I was looking for.
how to know the load generator configuration….?
I need a Best testcase template. Can u please share the same to my mail. rengarajan.m@sigmainfo.net
I fed up with ur online class. I paid the money but after two days of class I cant login into goto training session. The following message is displaying “The organizer has canceled your registration, so you cannot join this training session. Please contact the organizer with any questions.”
Hi,
Can STH, have date the article was published. Something published 4-6 years ago, might not make a good read today.
Kshitij.
@ Kshitij
The best way to find the date is to check the first comment :)
Really Nice article.
Shilpa, thanks for the article. I am not able to figure out the formula for the Cumultive Savings field in your ROI spreadsheet.
Could you please provide it to me?
Thank you!
Stephen
sfosterqa@gmail.com
very Useful Article. Thank you for sharing.
How to identify regression test cases in both manually or automation?
I am not able to figure out the formula for the Cumulative Savings field in your ROI spreadsheet.
Could you please provide it to me?
Thank you
Excellent Article. Thank you for sharing.
Thanks for sharing the useful information about the automation testing, keep sharing helpful blogs.
You have a great experience & knowledge & and will to share it.
I love what you’ve done here! Avoiding the trap where management is always only interested in reducing head count when hearing about automation.
Maybe it is time to start calling test automation something else and avoid that “automation” word.
Great blog!
A very detailed explanation about ROI seems to be realistic and concludes your dedication and hard work. This is great helpful for a software automation like me.
I need a Best testcase template. Can u please share the same to my mail. jan.malokela@suninternational.com
i be liking the automation testing. please do the needful and refer my profile. i do job like test or java just need $150,000 per year or higher. please let me know your bid, i start tomorrow.
i do the one with the selenium where we press the button with the automation machine each time. it does good on small and large applications when the timing does the stuff for consideration. uft can also do, if rational is no good. i have made the best testcase template for each of the testing, email me jan.malokela@suninternational.com
thankssssssssssssssssss :)
Hello!! This is great how can I get the ROI template?
How were the values for “Total automation effort (In hours)” in the ROI table calculated? Thanks!
Thank you so much for your amazing tips.
Hi, I will share my ideas here.
Usability Testing – at times this can be an impossible task to perform by automation as the computer cannot efficiently judge if the system is of any use to its users Tests that only need to be executed once – unless the same test needs to be executed for a large dataset then it makes sense to automate Tests without predictable results – test automation should give us confidence in the results of the tests.
If there are intermittent failures then the tests cannot be reliable and cannot be dependent on Tests that need to be verified visually Tests that need to be executed quickly. At first, writing an automated test takes longer. If we want a quick check, we should test manually, however, if that test is a good one which should be run regularly, then it should be automated in time