When To Opt For Automation Testing?

This article helps the reader to understand what Automation Testing is, when to go for it, and most importantly, when not to go for it. Also, learn optimal utilization of Automation Tools for Testing.

Testing will be carried out to provide good quality deliverables to the end user. Testing Phase is one of the main aspects of STLC.

Any company focuses more on software testing as its quality brings in optimal customer satisfaction, but many of them still struggle in choosing which kind of testing to carry out, either with automated testing or manual testing.

Automation Testing

Whatever work is done, it should be carried out effectively and must be cost-effective as well. Moreover, it should make sense so that the customer feels happy about the deliverables.

Software Testing and Cost Benefits

Software Testing is normally carried out by a Software Tester. The difference between a tester and a real user is that the latter will know only a partial usage of the software that is used for their business or for their tasks and will not know the software completely. On the other hand, a tester will be aware of all the technical and functional requirements of the software. Based on the requirements provided by the client, test plans and test cases will have to be prepared.

A test plan is nothing but a detailed plan of the way in which the testing process is to be carried out. This will have complete details about the number of resources & sources involved in testing, what to do & when to do it, what will not be done, and the environment in which it will be carried out etc.

Test cases should be prepared after a clear understanding of the functional and technical aspects of the software. The tester must possess a keen observational capacity and complete knowledge about the software.

Moreover, cost plays an effective role here. Customers prefer to accept software of maximum quality at a minimal cost. When we go for manual testing, the process is more tedious and time-consuming as it is all done manually by a tester.

For Example, when we need ‘n’ number of testers to execute Regression Testing, it may take nearly 50 hours to execute all the test cases. And based on the resource availability, the test cases will be executed. But with less time for automated testing, optimal utilization of resources is carried out along with the maximum coverage of test cases when compared to manual testing.

Intelligence Behind Software Testing

It is very important for any organization to know when to start the testing process and when to exit it. We should know when to start with testing because it’s useless to start testing when the development phase is done and when the criteria required are not met. It is always best practice to start with the test design phase while the development is in progress.

Given below are the criteria for software testing entry and exit:

Entry Criteria

Once the design document has been signed off, the test plans have to be prepared in the planning phase. A test plan plays a vital role. The hardware required must be installed and configured properly and the functionality of the hardware needs to be checked. Functional requirements must be cleared and approved. The developed code must be unit tested and signed off by the developers.

Test cases and test data must be prepared and approved. Test data and applications should be available. The tester must possess significant and sufficient knowledge on the application. Resources should be well trained about tools and must be clarified with all the required functionalities.

The tester should be available. When any one of the criteria is not attained, the entry criteria for testing is withheld.

[Note: Click on any image for an enlarged view]

Exit Criteria

Only when at least 95% of the mandatory test cases are locked with a “pass” result, we can exit the testing phase for the product. However, it’s not so easy to determine when software testing can be stopped or if it still needs to be executed. And this kind of situation commonly arises as well.

The main criteria are given below:

• Once all the bugs are fixed.
• When the deadline is reached.
• When the budget is exhausted or depleted.
• When all the test cases get passed.
• Once the agreement is signed off.
• When a certain percentage of testing is done.
• When the Alpha and Beta testing ends.

The exit criteria can be derived purely based on factors like risk, cost, etc. When testing of the main functional requirement has been achieved, then the testing will be stopped usually and they never look for minor bugs, which will create trouble in the later periods.

Example: Software ABC is in a design phase. Development and testing construction generally occur at the same time. After the design has been frozen, the development of the software starts. Completion of the development of the software, as agreed upon, indicates the entry criteria. Deliverables here are from the development team. This includes release notes and known issues.

After a few iterations of testing, when no major/blocker/show stoppers are pending resolution and 95% of testing have resulted in a pass, then it is referred to as exit criteria.

Automation – Is It Really Essential

When we need to decide if we require Automated Testing Technique or not, the question of available resources arises here.  The reason we need to automate this is in checking whether the data flow and the functionality developed is working as per the expectation without manual intervention. It is mainly used in places where the software will have changes in the form of multiple releases/cycles etc.

At the end of the development of each cycle, the testing of the currently added functionality will be done. Additionally, testing of the old functionality will be done to ensure that the old functionalities are not broken. This is a major part of the scope for automation.

When verifying the code-driven logic and the GUI requirements, one can choose Automated testing, provided the risk factor is high.

Example: For the Software ABC, there are frequent upgrades and updates being sought by the client and provided by the developers. Hence, as part of testing, regression is done for the software which is already live and running in production. Irrespective of any number of releases, upgrades, or updates, the current version is valid.

Say there are 10 days of manual efforts required for regression testing coverage, and then the utmost care for automating them must be taken. It can save at least 60% effort and 10*8 = 80 hours of manual work.

Automation can complete 80/24 = 3.33 days only.  This saves roughly 6.67.

Why Automation Testing

Automation can only be chosen when:

• The application is a very vast area with a high degree of investing effort in regression.
• Optimization in costs occurred due to manual errors.
• The software has multiple versions and releases.
• It is cost effective in the long run.
• The risk factor is higher for a broader scope of test execution.
• Cost figures and mathematical calculations are included in the software functionality.
• There is a greater increase in the execution tempo and efficiency along with the software quality.
• There is a lesser turn around time, even for high-risk software testing.

Risk Factor

Risk factors become pre-dominantly common in businesses where there are many dependencies on the time factor. Software that works based on transactional systems and that works across multiple applications will require the software to act ideally as per the software design. In this case, there are many risks involved in getting the correct functional behavior recorded.

Automation here will be very helpful in performing the functional transactions at a better pace as per the software mechanism.

For example, in the case of a Forex market indicator, the time factor is very important and critical. Changes in stocks and commodities occur with respect to time, sometimes less than seconds. Automation here can help in testing such software at high risk.

Example: Software ABC has multiple updates and upgrades. In order to save manual efforts and bring down the turn-around time for the testing phase, the base version, or the old functionalities, can be automated. This is valid only when the base functionalities remain unchanged.

The benefit of automation is that they can be run without any manual intervention. This can even be performed in parallel with testing newer functionality. Hence automation saves a lot of effort and time.

When should Automation not be PREFERRED

I have a question among several organizations which is – Why 100% Automation is not possible?

The answer from experts is NO because skilled users are required to carry out Automated testing and they must be well trained as well. Automation cannot be carried out during the initial stage of the criteria and requirements of the applications will not be clear.

Usually, automation is preferred from the second iteration of any software release. The user interface may be changed, which is more costly, and the script maintenance is also costlier. When the cost required for the automation tool exceeds the budget of the project we can say no.

Example: Software XYZ is a type of e-commerce site where the client requirements are not frozen and keep changing when required by the clients.

Here in this case, automation cannot help the regression. This is because the old functionalities that are not valid should not be tested, and therefore, they have to be done manually. For instance, a client needs to have all list boxes in the base software to be changed as drop-down boxes.

Cost vs ROI for Automation

The ROI is very low when we go for automation because automation is expensive for the first time. ROI keeps increasing as the manual effort in testing the software lowers from the iterations of the second release. We need to be aware of the expected outcome of any test cases prior to Automation.

Consider the design of the test cases more important when choosing Automation and any tools to ensure they will not increase the cost.

Where can Automation render minimal to NO cost REDUCTION

Even automation costs because the required tool for testing has to be purchased. Resources have to be trained with a particular tool. The tool chosen must be feasible to test all the areas of the software.

So the tool selection should be done carefully by the experts of Automation testing.

Example: Consider product XYZ which deals with insurance. To reduce the cost factor, the company used only manual testing, but when it comes to insurance, the risk factor is high and may cost the firm money when any one of the premium calculation goes wrong. The entire loss will be either for management or for the end user. The end user will not bear loss whereas the company has to.

If the calculated premium amount mismatches with the original premium (i.e.) when there is a difference in the front end and back end premium calculation, then a great problem arises between the customer and the product seller. It may contain many modules like automobiles, home and others as well.

When anything goes wrong, it is a complete loss. The difference in the calculation can make sense to the tester and may raise bugs. In this project, manual testing can be done for the basic UI such as verifying the TIN number, social ID and other information related to the user portfolio and hence can be tested manually where the risk factor is low. The more the company would profit, the more they prefer automation for testing their software.

Conclusion

Both automation and manual testing have advantages and disadvantages as well. Only when we are clear about the concepts and the requirements will we be able to choose what kind of testing to carry out.

No project can be tested with manual testing or automated testing alone. It depends upon the design, platform, and the technology with which the software has been developed. So, when making a decision, one must be careful in choosing the method of testing and use the advice of experts.

In the above article, we may have missed a few factors, kindly share the factors which you think are important while choosing automation or even tools for automation.

Also, feel free to share your comments/suggestions about this article. We would love to hear from you.