How to Choose the Best Automation Testing Tool (A Complete Guide)

In this tutorial, we will learn how to choose the best automation tool. We have covered the test automation tool selection criteria and checklist with the test automation tool comparison matrix for your reference.

A to Z Guide on Selecting the Best Automation Tool for Your Project

This is the 4^th tutorial in our Test Automation Tutorial series. You can check all the articles posted in this series by clicking on this link: => The Ultimate Guide to Start Automation Testing on Your Project

Test automation tool selection is one of the most important steps before starting automation in any organization. This is important because the tool will affect your entire automation effort.

Choosing an Ideal Automation Tool

The automation becomes easier and more effective if the tool is good and gives you the required features.

There are a variety of criteria to consider while selecting the automation tool. I have discussed this in one of my previous articles. I have listed some of the most important aspects to consider while selecting the test automation tool.

Is Automated Testing a Solution for You

I’ve worked on multiple projects in my career. After working on the same project for more than a year, you feel a need to automate some tasks. You can start thinking about introducing automation testing on the project if project management has not considered it till now.

One year is enough time for anyone to know the entirety of any project. Once you know the project functionality, it becomes easier to decide which repetitive tasks need to be automated.

Some testers also get bored doing the same tasks again and again and they strongly start feeling the need for test automation.

Does this mean you should jump into automation testing right away? I don’t agree. There are several criteria you need to work on before deciding whether automation is a solution for you.

When is Test Automation a Logical Choice

These are a few potential criteria to consider.

• When there are many repetitive tests
• When there are frequent regression testing iterations
• If you need to simulate a large number of users who are using the application resources
• When AUT is having a comparatively stable UI
• If you have a large set of BVT cases
• If you can’t rely solely on manual test execution for critical functionality

Further Reading => When should you go for automation?

Once you know it is the right time to invest your time and money in a good automation tool, you can start looking for the best automation tool that suits your requirements.

Also Read =>> Tips you read should before starting automated testing

How to Select an Automation Tool for Your Project

Automation testing success largely depends on the selection of the right testing tools. It takes a lot of time to evaluate the relevant automation tools available in the market. However, this is a compulsory onetime exercise that will benefit your project in the long run.

There were a few situations where I got the chance to review and select automation tools for my projects. Despite the challenging task of balancing testing needs and cost restrictions, the experience was valuable.

Test Automation Tool Evaluation Criteria

Here are the criteria you need to consider before selecting any testing tool:

You may also want to consider other criteria, such as:

Tips: Requirement gathering is by far the most important step for selecting the right tool. Categorize your requirements in must-have, nice-to-have, and not required feature categories. This will help you evaluate the tool quickly. Remember, you won’t find a tool already available in the market that will support all your automation needs!

HP QTP/UFT and Selenium are the two most popular functional testing options available currently. QTP/UFT is the best functional testing tool supported by a wide range of coding languages and platforms, whereas Selenium is the best open-source functional web testing tool.

Read the article mentioned below for the list of TOP tools

In the next article, we’ll discuss the manual and automation testing challenges.

Test Automation Tool Selection Criteria and Checklist

10 Questions to Ask Before Selecting the Best Automation Testing Tool

Ask the following questions whenever you are in a situation to select the automation tool for your organization.

Question #1: What is the budget of your organization for automation tools?

This is the most important thing to consider while selecting the automation tool.

Why look for QTP/UFT and research on it when you cannot purchase the license? The QTP tool costs around $8000 (Approx.). If your organization can purchase the license and you are confirmed, then you should download the trial and make a pivot automation project on it to test its feature.

Otherwise, you should not spend time researching it. (I am talking about this scenario if you want to use QTP on a live project of the company. If you are downloading it just for learning purposes, then it is OK to download the trial.)

Question #2: What is the actual price of the tool?

Next is the price of the automation tool. There is not only a license price but also the price of add-ons (if needed), the support fee, the training fee, and the upgrade fee.

Let’s talk about the license first.

a) Types of Licenses

There are the following types of licenses:

Node-Locked User License: The Node-Locked User license will support a test automation tool to use on a single physical computer in your company network. You can only run one instance of the tool on the licensed computer at a time. This license is usually bound to the machine’s hostname.

Concurrent Floating User License: A floating user license can be shared between different machines, but can only be used by one machine at a time. It is not bound to a machine name or anything, instead, it uses a license manager (installed on a server) to manage the same license across various machines.

Basically, with the Node-Locked license, you don’t have the liberty to install the tool on one machine, uninstall it, and then install it again on any other machine. However, with the Floating user license, you are allowed to do that.

Run Time License: The above-mentioned two types of licenses are usually bought to “develop” the scripts. So these are “development” Licenses. To execute the scripts on different machines, you need to have the “execution” or “Runtime” license for each machine.

For example, if a tester needs to develop and execute test cases on the same machine, then one development license is enough. But if he needs to develop on one machine and execute the test cases on three different virtual or physical machines, he needs to buy one “development” license and three runtime licenses.

Some vendors offer free runtime licenses (Like Coded UI) and some offer a price (like Test Complete, Ranorex, etc). So it all depends from vendor to vendor.

Open Source License: It is your company’s choice to go for a commercial tool and pay a cost or go for an open source tool.

Commercial tools are expensive, but they offer great support and are easy to use with lots of training material provided. Commercial tools are usually “one tool for all needs”. Open-source tools are free but are generally harder to learn. There is very little official support, but you can find solutions by visiting different forums. Open-source solutions are normally used for specific needs.

b) Support, Upgrade and Training Fee

For support, training, and an upgrade fee, you may need to call the company representative. Some companies offer special discounts on bulk purchases of licenses, so sometimes this information is not mentioned on websites. You will receive the information through call or email only.

Question #3: Does the tool support the Operating System/Browser or Device in which your application runs?

This question is normally dependent on the type of application you are using.

a) If Desktop Based

If you are working on a desktop application, you should outline how many operating systems you want to test that application. I was working on a desktop-based application and I wanted to test it on Windows 7 and Windows 8.1. I chose Coded UI because it supports both.

b) If Browser Based

If you are working on a web application, you should outline on how many browsers you want to test this application. I wanted to execute my test cases on Firefox, Chrome, and IE. I chose Selenium for my web application because it supports all these browsers. Make sure that the tool you choose should support both older as well as new versions of your required browsers.

c) If Mobile Based

If you are working on Mobile applications, you should know which mobile operating systems you have to run your test cases. If your application runs on both Android and IOS, your tool should support that. Selenium has separate drivers to run scripts on Android, IOS, Windows Phone, and BlackBerry.

You can also use a separate tool for each of the Mobile OS. There is Robotium for Android, Appium for both IOS and Android, and CodedUI for Windows phone apps.

Again, this comes amidst the debate between open source vs commercial. As you can see there are separate open-source tools to test web-based, mobile based, and desktop-based applications.

But if you go for a commercial tool like Test Complete, Ranorex, or Test Studio, they can test all three types (Mobile, Desktop, and Browser-Based Applications). So in the case of the commercial tool, you need to learn only one tool to test web, desktop, and mobile applications.

Question #4: Does the tool support the technologies and third-party controls used in your application?

This is a very important aspect of selecting the tool. You should know firsthand what technologies are being used in your application. Consult your developers and write these down. If they are using HTML 5 or SilverLight in web applications, beware, that there are not many automation tools to support them.

If the tool claims support for these technologies, download the trial version of the tool and try to identify different objects in your application. If the tool fails to identify them, then their claim is false. This activity will save you from the aftermath of misery.

Test Automation Tool Comparison Matrix

The following table compares different tools concerning their licensing price and their support for different technologies. (You should use this chart as a learning practice in how to create comparisons between different tools, but the accuracy of the data given is not 100%)

(Click on the image for an enlarged view)

Y = Supported, N = Not Supported, U = Unknown

Question #5: How many languages does the tool support? Do you have skilled resources for these languages?

Learning the tool is one thing. But learning a language is another aspect altogether. If you have resources that have expertise in Java, and your tool does not support Java, the time to learn the new language will be added to your automation effort.

Another aspect is that if your product is built on Java, you must have a team of developers who are experts in Java. These developers can also help the automation team in terms of language-related issues. Selecting the tool which offers a language that is familiar to your resources is important and it will help you to minimize the learning curve for your resources.

The Selenium WebDriver offers writing scripts in multiple languages such as C#, Java, Python, Ruby, and in JavaScript. TestComplete also offers writing scripts in multiple scripting languages such as VBScript, JScript, DelphiScript, C++Script, and C#Script.

Question #6: Does the tool support connecting to different Data Sources?

If we are using an automation framework like keyword-driven or data-driven, we need to have the ability to connect our tool to any data source. If the tool provides connectivity with different data sources easily, it will be very beneficial.

See the support for common data sources such as a CSV file, Excel Files, XML files, and Databases. If these are present in the tool, you are good to go.

Question #7: How is the reporting mechanism of the automation tool?

When we execute the script, it will either pass or fail. In the case of the pass, there is not much information needed except for the duration and environment info. But in case of failure, we need a comprehensive report about the failure. The report should tell us exactly on which step the script fails. A snapshot of the moment of failure will be an added advantage.

This report should also be exported to different formats so that we can share this with stakeholders. Many of these options are built-in and in some tools, there are ways to make your report comprehensive. This is another thing to look out for when you download the trial version of the tool. If it is giving comprehensive reports on failures, it’s best for the organization.

Question #8: Can the tool be integrated with test case and bug management repositories?

There is a good chance that your organization is already using any test case or bug management tool. Companies want their automated tool to be integrated with their existing test case management tool so that their whole application lifecycle is effectively managed. This aspect should also be seen while selecting the test automation tool.

QTP supports QLM, Coded UI supports TFS and TestComplete supports QAComplete. Some open-source tools also have support to integrate with existing open-source test management tools. It all depends on what your organization is using.

Question #9: How is official technical support for the tool?

Here we are only talking about commercial tools. When you select a commercial tool, its support aspect is very important. See the training material provided on the website. Does the website contain videos and tutorials? Does the website have an official forum to ask questions? Download the trial, shoot a question on their forum and see how many days it gets answered. Do they provide support for a call?

The above questions should be asked every time because you are spending a good amount of money on the tool. If the tool does not have good support, don’t buy it.

Question #10: Some Technical Aspects to see

There are a few other technical aspects to see as well such as the following:

a) Record and playback support

It is not a recommended approach in test automation, but it is good to have a tool. It simplifies the learning process of the tool and helps easy scenarios to be easily automated.

b) Different Object Recognition methods and Object Mapping Support

There should be a variety of selecting the same object with different methods. Some objects are difficult to recognize. So the variety of selection methods is always helpful.

For example, selenium supports selecting objects by id, name, class, link test, XPATH, CSS selector and JavaScript. Here is a tutorial on – how QTP identify objects uniquely. If one selection method is not working, we have a variety of other methods to choose from which is always helpful.

Similarly, there should be an option to properly map these objects in the object repository. This repository should be easily updateable and managed. Just to remind you that Selenium does not have built-in support for object mapping.

c) Different Checkpoints or Assertion Support

Test cases are passed or failed based on checkpoints or assertions. If the tool has a variety of methods to check your expected results, it is beneficial. QTP has a variety of checkpoints such as Standard, Bitmap, Table, XML, Database, and File Content Check Points.

d) Recovery Scenarios Handling

If the test case fails and you want to continue execution, does the tool support that easily? If recovery scenarios are easy to manage in a tool, it will allow you to execute test cases without any glitches. You can run the test cases at night and in the morning you get the results stating which test cases are failed and which test cases are passed.

This will only happen if recovery from failed test cases can be easily managed by the tool. Otherwise, a good amount of automation effort will be wasted in handling recovery scenarios. See recovery scenarios management in QTP.

Conclusion

Always remember that no tool can be classified as good or bad. It all depends on your requirements and product nature.

Selenium may be the most popular automation tool, but if your product is desktop-based, this tool has no use for you. Understand your product and then search for the appropriate tool which matches your requirements using the guidelines mentioned in this tutorial.

Correct automation tool selection plays a vital role in successful automation.

Next tutorial – Our next tutorial in this series is on ‘Script development and automation frameworks with examples’. Again, check out all the tutorials in this series on this page.

Feel free to post your queries/feedback in the comments section below about selecting the right automation tool. We would love to hear from you. 

PREV Tutorial #3 | NEXT Tutorial #5