How to Choose the Best Automation Testing Tool (A Complete Guide)

In this tutorial, we have covered test automation tool selection criteria and checklist with test automation tools comparison matrix for your easy reference.

The A to Z Guide on Selecting the Best Automation Tool for Your Project:

This is the 4^th tutorial in our Test Automation Tutorial series. Please check all articles posted in this series on this page: => The Ultimate Guide to Start Automation Testing on Your Project

Test automation tool selection is one of the most important steps before starting automation in any organization. 

It is important because the tool will greatly affect your whole automation effort. If the tool is good and gives you required features, the automation becomes easier and effective.

There are many criteria to consider while selecting the automation tool. Some of them I have discussed in one of my previous articles. Here I have listed down the most important aspects to consider while selecting the test automation tool.

Is Automated Testing a Solution for You?

I’ve worked on many projects in my career. When you work on the same project for more than one year you strongly start feeling a need for automating some tasks. You start thinking to introduce automation testing on the project if it’s not been considered till now by project management.

One year is enough time for anyone to know the ins and outs of any project. Once you know the project functionality in detail it becomes easier to decide which repetitive tasks need to be automated.

Some testers also get bored

doing same repetitive tasks again and again and they strongly start feeling the need for test automation.

Does that mean you should jump into automation testing right away?

Definitely not!

There are many criteria you need to work on before deciding whether automation is a solution for you.

When Does Test Automation Make Sense?

• When there are many repetitive tests
• When there are frequent regression testing iterations
• When you need to simulate a large number of users who are using the application resources
• When AUT is having comparatively stable UI
• When you have a large set of BVT cases
• When you can’t rely solely on manual test execution for critical functionality

Further reading:

• When should you go for automation?
• Tips you read should before starting automated testing

Once you know that it is the right time to invest your time and money in a good automation tool you can start looking for the best automation tool that suite your needs.

How to Select Automation Tool for Your Project?

Automation testing success largely depends on the selection of right testing tools. It takes a lot of time to evaluate relevant automation tools available in the market. But this is a must one-time exercise that will benefit your project in long run.

There were few situations where I got the chance to review and select automation tool for my projects. The task was difficult as we had to manage our testing needs and cost restrictions but it was a worth experience.

Here are the criteria you need to consider before selecting any testing tool:

Test Automation Tool Evaluation Criteria

1) Do you have the necessary skilled resource to allocate for automation tasks?

2) What is your budget?

3) Does the tool satisfy your testing needs? Is it suitable for the project environment and technology you are using? Does it support all tools and objects used in the code? Sometime you may get stuck for small tests due to inabilities of the tool to identify the objects used in the application.

I consider above three factors as most important for selecting any tool.

4) Does the tool provide you the free trial version so that you can evaluate it before making a decision? Also, does the tool have all features available in the trial version?

5) Is the current tool version stable? Is the vendor company established with good customer support as well as online help resources and user manual?

6) How is the tool learning curve? Is the learning time acceptable for your goals?

7) Do you want automation tool for only your project needs or you are looking for a common tool for all projects in your company? It would be a good choice if you select a tool that supports most of the coding languages on your projects.

8 ) Which testing types does it support? A tool which supports maximum testing types (Unit, functional, regression etc.) is always a better choice. Warning – Don’t go for a tool just because it is supporting all testing types. It’s also important that the tool should be powerful enough to automate your complex requirements.

9) Does the tool support easy interface to create and maintain test scripts? Record and playback tool with abilities to edit recorded scripts could be a good solution.

10)  Does it provide simple interface yet powerful features to accomplish complex tasks?

11)  How easy is it to provide input test data for complex or load tests? A tool supporting test data input from various data files such as Excel, XML, text file etc. would be a big relief for the automation the testers.

12)  Does it provide the powerful reporting with graphical interface? Clear and concise reports will always help you to conclude the test results quickly.

13)  Does it integrate well with your other testing tools like project planning and test management tools?

You may also want to consider other criteria like:

14)  Tool vendor refund policy

15)  Existing customer reviews for the tool

16)  Is the vendor providing initial training?

Tips: Requirement gathering is by far the most important step for selecting the right tool. Make sure to categorize your requirements in must have, nice to have, and not required feature categories. This will help you to evaluate the tool quickly. Remember you won’t find a tool already available in the market which will support all your automation needs!

Best Automation Tools:

HP QTP/UFT and Selenium are the two most popular functional testing options available currently. QTP/UFT is a best functional testing tool supported on the wide range of coding languages and platforms whereas Selenium is the best open source functional web testing tool.

Read this article for the list of TOP tools:

Top 20 Best Automation Testing Tools In 2020 (Comprehensive List)

In next article, we’ll discuss manual and automation testing challenges.

Test Automation Tool Selection Criteria and Checklist

10 Questions to Ask Before Selecting the Best Automation Testing Tool

Ask the following questions whenever you are in a situation to select the automation tool for your organization:

Question #1: What is the Budget of your organization for automation tool?

This is, in my opinion, the most important thing to consider while selecting the automation tool.

Why look for QTP/UFT and research on it when you cannot purchase the license? The QTP tool costs around $8000 (Approx.). If your organization can purchase the license and you are confirmed, then you should download the trial and make a pivot automation project on it to test its feature. Otherwise, you should not spend time researching on it. (I am talking about this scenario if you want to use QTP on a live project of the company. If you are downloading it just for learning purposes, then it is OK to download the trial.)

Question #2: What is the actual Price of the tool?

Next is the price of automation tool. There is not only a license price but also the price of add-ons (if needed), the support fee, the training fee and upgrade fee.

Let’s talk about the license first.

a) Types of Licenses:

There are following types of licenses.

1) Node-Locked User License.

The Node-locked User license will support test automation tool to use on a single physical computer in your company network. You can only run one instance of the tool on the licensed computer at a time. This license is usually bound to machine’s hostname.

2) Concurrent Floating User License

A floating user license can be shared between different machines, but can only be used by one machine at a time. It is not bound to machine name or anything, instead, it uses a license manager (installed on a server) to manage the same license across different machines.

Basically, with the Node-Locked license, you don’t have the liberty to install the tool on one machine, uninstall it and then install it again on any other machine. But with Floating user license, you are allowed to do that.

3) Run Time License

The above mentioned two types of licenses are usually bought to “develop” the scripts. So these are “development” Licenses. To execute the scripts on different machines, you need to have the “execution” or “Runtime” license for each machine.

Example:

For example, if a tester needs to develop and execute test cases on the same machine, then one development license is enough.

But if he needs to develop on one machine and execute the test cases on three different virtual or physical machines, he needs to buy one “development” license and three runtime licenses.

Some vendors offer free runtime licenses (Like Coded UI) and some offer with a price (like Test Complete, Ranorex etc.). So it all depends on vendor to vendor.

4) Open Source License

It is your company’s choice to go for a commercial tool and pay a cost or go for an open source tool.

Commercial tools are expensive, but they offer great support and are easy to use with lots of training material provided. Commercial tools are usually “one tool for all needs”. Open source tools are free but are generally harder to learn. There is little official support, but you can find solutions by visiting different forums. Open source solutions are normally for specific needs.

b) Support, Upgrade and Training Fee:

For support, training and an upgrade fee, you may need to call the company representative. Some companies offer special discounts on bulk purchase of licenses, so sometimes this information is not clearly mentioned on websites. You will get the information through call or emails only.

Question #3: Does the tool support the Operating System/Browser or Device in which your application runs?

This question is normally dependent on the type of application you are using.

a) If Desktop Based:

If you are working on a desktop application, you should outline that on how many operating systems you want to test that application. I was working on a desktop based application and I wanted to test it on Windows 7 and Windows 8.1. So I chose Coded UI because it supports both.

b) If Browser Based

If you are working on a web application, you should outline that on how many browsers you want to test this application. I wanted to execute my test cases on FireFox, Chrome, and IE. I chose selenium for my web application because it supports all these browsers. Do make sure that the tool you choose should support both older and newer versions of your required browsers.

c) If Mobile Based

If you are working on Mobile applications, you should know that on which mobile operating systems you have to run your test cases. If your application runs on both Android and IOS, your tool should support that. Selenium has separate drivers to run scripts on Android, IOS, Windows Phone, and BlackBerry. You can also use a separate tool for each of the Mobile OS. There is Robotium for Android, Appium for both IOS and Android and CodedUI for Windows phone apps.

Again, this comes to the debate of open source vs commercial. As you can see there are separate open source tools to test web-based, mobile based and desktop based applications. But if you go for a commercial tool like Test complete, Ranorex or Test Studio, they can test all three types (Mobile, Desktop, and Browser-Based Applications). So in case of the commercial tool, you need to learn only one tool to test web, desktop, and mobile applications.

Question #4: Does the tool support the technologies and third-party controls used in your application?

This is a very important aspect while selecting the tool. You should know first hand that what technologies are being used in your application. Consult your developers and write these down. If they are using HTML 5 or SilverLight in web applications, beware, there are not many automation tools to support them. If the tool claims support for these technologies, download the trial version of that tool and try to identify different objects in your application. If the tool fails to identify them, then their claim is false. That activity will save you from the afterward misery.

Test automation tools comparison matrix:

The Following table compares different tools with respect to their licensing price and their support for different technologies. (You should take this chart as a learning practice in how to create comparisons between different tools, but the accuracy of the data given is not 100%)

(Click on image to view enlarged)

Y = Supported, N =Not Supported, U = Unknown

Question #5: How many languages the tool supports? Do you have skilled resources for these languages?

Learning the tool is one aspect. Learning the language is another aspect. If you have resources that have expertise in Java, and your tool does not support Java, the time to learn the new language will be added to your automation effort.

Another aspect is that if your product is built on Java, you must have a team of developers that are experts in Java. These developers can also help the automation team in terms of Language related issues. Selecting the tool which offers a language which is familiar with your resources is important and it will help you to minimize the learning curve for your resources.

The Selenium WebDriver offers writing scripts in multiple languages such as C#, Java, Python, Ruby and in JavaScript. TestComplete also offers writing script in multiple scripting languages such as VBScript, JScript, DelphiScript, C++Script, and C#Script.

Question #6: Does the tool support connecting to different Data Sources?

If we are using an automation framework like keyword driven or data-driven, we need to have the ability to connect our tool to any data source. If the tool provides connectivity with different data sources easily, it will be very beneficial.

See the support for common data sources such as a CSV file, Excel File, XML file, and Database. If these are present in a tool, you are good to go.

Question #7: How is the reporting mechanism of the automation tool?

When we execute the script, it will either pass or fail. In case of the pass, there is not much information needed except for the duration and environment info. But in case of failure, we need a comprehensive report about the failure. The report should tell us that exactly on which step the script fails. A snapshot of the moment of failure will be an added advantage.

Also, this report should be exported to different formats so that we can share this with stakeholders. In many tools, these options are built-in and in some tools, there are ways to make your report comprehensive. This is another thing to look out when you download the trial version of the tool. If it is giving comprehensive reports on failures, it’s best for the organization.

Question #8: Can the tool be integrated with test case and bug management repositories?

There is a good chance that your organization is already using any test case or bug management tool. Companies obviously want their automated tool to be integrated with their existing test case management tool so that their whole application lifecycle is managed properly. This aspect should also be seen while selecting the test automation tool.

QTP supports QLM, Coded UI supports TFS and TestComplete supports QAComplete. Some Open source tools also have support to integrate with existing open source test management tools. It all depends on what your organization is actually using.

Question #9: How is official technical support for the tool?

Here we are talking about commercial tools only. When you select a commercial tool, their support aspect is very important. See the training material provided on the website. Does the website contain videos and tutorials? Does the website have an official forum to ask questions? Download the trial and shoot a question on their forum and see how many days it gets answered. Do they provide support for a call?

The above questions should really be asked every time because you are spending a good amount of money on the tool. If the tool does not have good support, don’t bother to buy it.

Question #10: Some Technical Aspects to see

There are some other technical aspects to see as well such as:

a) Record and playback support

It is not a recommended approach in test automation, but it is good to have a tool. It simplifies the learning process of the tool and helps easy scenarios to be easily automated.

b) Different Object Recognition methods and Object Mapping Support

There should be a variety of selecting the same object with different methods. Some objects are difficult to recognize. So the variety of selection methods are always helpful. For example, selenium supports selecting objects by id, name, class, link test, XPATH, CSS selector and JavaScript. Here is a tutorial on – how QTP identify objects uniquely. If one selection method is not working, we have a variety of other methods to choose from which is always helpful.

Similarly, there should be an option to properly Map these objects in the object repository. This repository should be easily update-able and managed. Just to remind you that Selenium does not have built-in support for object mapping.

c) Different Checkpoints or Assertion Support.

The test case is passed or failed based on checkpoints or assertions. If the tool has a variety of methods to check your expected results, it is beneficial. QTP has a variety of checkpoints such as Standard, Bitmap, Table, XML, Database and File Content Check Points.

d) Recovery Scenarios Handling.

If the test case fails and you want to continue execution, does the tool support that easily? If recovery scenarios are easy to manage in a tool, it will allow you to execute test cases without any glitch. You can run the test cases at night and in the morning you get the results stating which test cases are failed and which test cases are passed. This will happen only if recovery from failed test cases can be easily managed by the tool. Otherwise, a good amount of automation effort will be wasted in handling recovery scenarios. See recovery scenarios management in QTP.

Conclusion

Always remember that no tool is a good tool or bad tool. It all depends on your requirements and product nature.

Selenium may be the most popular automation tool, but if your product is desktop based, this tool has no use for you. Understand your product first and then search for the appropriate tool which matches your requirement using the guidelines mentioned in this tutorial.

Correct automation tool selection plays a vital role in successful automation.

Next tutorial – Our next tutorial in this series is on ‘Script development and automation frameworks with examples’. Again, check all tutorials in this series on this page.

Feel free to post your queries/comments below about selecting the right automation tool.

PREV Tutorial #3 | NEXT Tutorial #5