How to Select Correct Test Cases for Automation Testing (and Ultimately Achieve a Positive Automation ROI)

Now is the era of test automation. Most of the testing projects are trying to translate their manual test cases to automated one to improve productivity and coverage.

One of the key steps to commence automation testing is – selecting the appropriate test cases and determining the ROI (Return on Investment).

What to expect from this article?
In this article, I have tried to cite down few important points based on my experience to help you select the correct candidate of automation and determine various other factors which will yield better test results and benefits.

Why Automated Testing?

Automation does not overpower or replaces manual testing but it compliments it. Like manual, automation to needs a strategy with proper planning, monitoring & control. Automation, when implemented correctly, can become an asset to the team, project and ultimately to the organization.

There are many advantages of automation; here are few important to mention:

• Useful to execute the routine tasks like smoke tests and regression tests.
• Useful in preparing the test data.
• Helps to execute the test cases which involve complex business logic.
• Good to execute the cross-platform test cases (like different OS, browsers etc.)
• Great to execute the test cases which are a bit difficult to execute manually.
• When the number of iterations of the test case executions is not known.

Many a time stakeholders feel that test automation acts as a support tool for manual testing, so it’s vital to understand that automation is the best way to increase the effectiveness, efficiency and coverage of testing. It not only saves time but also improves accuracy as repetitive tasks via manual approach can prone to human errors and can be time-consuming.

Candidates for Automation

Basic Mistake to Avoid:
One of the most basic mistakes which testers make is NOT Selecting the correct test cases for automation.

Don’t just select any test suite. Analyze the test cases thoroughly and select the candidates for automation considering the most important factor i.e. ROI. First, we should understand and find out the ways for getting a higher and positive ROI.

(ROI – Return on Investment – It is a calculation of benefits in terms of cost saving, increased efficiency and quality)

There is no standard procedure for determining the correct test cases for automation. It all depends on the application you are testing.

Based on my experience, I have tried to chalk down few steps which may provide some insight for selecting the test cases and ultimately progress to achieve a positive ROI for automation.

See Also = > How to Translate Manual Test Cases into Automation Scripts?

How to Select Correct Test Cases for Automation Testing

Step 1:

Identify the parameters on which you will base your test case as a candidate for automation.

As of now, I am identifying the below parameters, you can have your own parameters depending on your application.

• Test case executed with different set of data
• Test case executed with different browser
• Test case executed with different environment
• Test case executed with complex business logic
• Test case executed with different set of users
• Test case Involves large amount of data
• Test case has any dependency
• Test case requires Special data

Step 2:

Break each application into modules. For each module, analyze and try to identify the test cases which should be automated based on the parameters. This list will vary for projects to projects and can also be enhanced to suit your needs:

Fig 1.0 (click on image to enlarge view)

Y – Yes
N – No

In a similar way, for all modules, this list can be used to identify the automation candidate test cases.

Step 3:

Consolidate and group the number of test cases for each module shown below

Fig 2.0 (click on image to enlarge view)

Figure 2.0 is quite straightforward and self-explanatory. Here I am trying to quantify the details and giving an estimate to finish the testing manually.

Step 4:

Once you have identified all the granular level details, you can present them in the below way. We are now progressing to calculate the ROI.

Fig 3.0:

We should also take into account the below attributes which form the basis of deterring the ROI:

• Purchasing and licensing cost of the tool
• Time to develop the scripts
• Time to maintain the scripts.
• Time to analyze the results manually and automatically
• Time and cost to train the resources.
• Management overheads

Test Automation ROI Calculation Example:

In most of the cases, ROI is calculated for 5 years, but it’s not mandatory. Based on the above factors, let me try to elaborate the calculation of ROI for 5 years. As usual, you can always tailor and enhance it.

(click on image to enlarge view)

*ROI = (Cumulative saving / Investment through automation)*100

Manual to Automation Testing – What are the Process Challenges?

I have tried to cite the points, which I feel, are a big challenge, when we try to automate a test suite.

#1. Automation Need: Every test team is unique and has an exclusive need for automation. We cannot develop a fixed standard, but we can tailor a standard which will suit our need.  Because of this reason, automation does require a good support from the management as well as from the development team.

#2. Automating the complete application: Automating 100% application is a big task. Not that it is impossible, but it requires proper planning and monitoring and of course; some time. There are lots of permutations and combinations of data, n number of environments with n number of authentication and authorization attributes which needs to be validated and hence requires a strategy to automate.

#3. Manual Vs Automation mentality: “We normally automate which is important and repetitive, but we prefer to test the important functionality manually”. Confused? Even I am!! But this is a fact. We should have criteria which will decide which are the important test cases. These criteria can be based on multiple factors like complex business logic, areas which are of more interest to clients, risk-prone areas etc.

#4. Deciding on the framework: Designing the framework is the most important facet of automation. I believe we should devote relatively more time to develop the framework than to script. Whenever we develop the automation plan, framework designing should be the main focus. PLAN to design the framework. Identify and make a checklist of the items which will form the framework. If the framework is rock solid, scripting and maintaining become easy.

#5. Knowledge of the team: Whenever we think of automation, we immediately jump to learn the programming language or scripting language. Learning this language will definitely help but more emphasis should be on building and developing logic. Automation should not be the responsibility of some handful resources, rather an entire team should contribute towards it. This will help not only to enhance the skills of the resources but also keep them motivated.

#6. Reporting: Every tool has a standard to report the test results. To customize it; is a challenging task. Reporting the test results also requires coordination and maintenance which adds to the cost.

#7. Trust: We should trust our automation stuff. We invest man-hours to build an automation suite but still, we don’t believe in the test results. Efforts should be given to maintaining the scripts. Also, we should see that the team who is doing the manual testing of the application, should be involved to automate it as they know their application. Most of the time, a third team does the automation so the actual testing team is not aware of the scripts and finally end up with running the tests manually because they feel follow up on the scripts,  add up to their tasks

See Also => Manual and Automation Testing Challenges.

Conclusion:

In most of the cases, we prefer to automate the regression suite (here are some challenges in automating regression suite in an agile environment) as it contains a bigger number of test cases. In that case, we can break down the regression suits into smaller suits and decide to run the appropriate suite as per the release requirement. Suppose a regression suite contains 1500 test cases, you can break it to 3 suits of 500 test cases per suit and automate it.

Instead of automating the entire suite, you can opt for phase-wise automation. In other words, you can follow the prototype model for developing the automation suite. Create a structure or framework with an implementation of fewer numbers of test cases and start using that and gradually enhance it by adding more test cases to it.

We should follow the Deming wheel (PDCA cycle) for automation as well. Being an ongoing activity, emphasis should be laid on building the framework properly which will result in ease of maintenance and implementing new features. It requires proper support from the dev team and from management as well. We should encourage the testing team to contribute the most for automation testing as they know their product more than anyone else.

About the Author: This is a guest article by Shilpa Chatterjee Roy. She is working in software testing field for the past 8.5 years in various domains.

Hope we have simplified this ‘manual to automation testing’ process. Please feel free to share your experience and thoughts on how you overcome the process challenges.