This is 4th tutorial in our Test Automation Tutorial series. Please check all articles posted in this series on this page: => The Ultimate Guide to Start Automation Testing on Your Project
Test automation tool selection is one of the most important steps before starting automation in any organization. It is important because the tool will greatly affect your whole automation effort. If the tool is good and gives you required features, the automation becomes easier and effective.
There are many things to consider while selecting the automation tool. Some of them I have discussed in one of my previous articles. Here I have listed down the most important aspects to consider while selecting the test automation tool.
Ask the following questions whenever you are in a situation to select the automation tool for your organization:
What You Will Learn:
- 10 Questions to Ask Before Selecting the Best Automation Testing Tool
- Question #1: What is the Budget of your organization for automation tool?
- Question #2: What is the actual Price of the tool?
- Question #3: Does the tool support the Operating System/Browser or Device in which your application runs?
- Question #4: Does the tool support the technologies and third party controls used in your application?
- Question #5: How many languages the tool supports? Do you have skilled resources for these languages?
- Question #6: Does the tool support connecting with different Data Sources?
- Question #7: How is the reporting mechanism of the automation tool?
- Question #8: Can the tool be integrated with test case and bug management repositories?
- Question #9: How is official technical support of the tool?
- Question #10: Some Technical Aspects to see:
10 Questions to Ask Before Selecting the Best Automation Testing Tool
Question #1: What is the Budget of your organization for automation tool?
This is, in my opinion, the most important thing to consider while selecting the automation tool.
Why look for QTP/UFT and research on it when you cannot purchase the license? The QTP tool costs around $8000 (Approx.). If your organization can purchase the license and you are confirmed, then you should download the trial and make a pivot automation project on it to test its feature. Otherwise, you should not spend time researching on it. (I am talking about this scenario if you want to use QTP on a live project of the company. If you are downloading it just for learning purposes, then it is OK to download the trial.)
Question #2: What is the actual Price of the tool?
Next is the price of automation tool. There is not only a license price, but also the price of add-ons (if needed), the support fee, the training fee and upgrade fee.
Let’s talk about the license first.
a) Types of Licenses:
There are following types of licenses.
1) Node-Locked User License.
The Node-locked User license will support test automation tool to use on a single physical computer in your company network. You can only run one instance of the tool on the licensed computer at a time. This license is usually bound to machine’s host name.
2) Concurrent Floating User License
A floating user license can be shared between different machines, but can only be used by one machine at a time. It is not bound to machine name or anything, instead it uses a license manager (installed on a server) to manage the same license across different machines.
Basically, with Node-Locked license, you don’t have the liberty to install the tool on one machine, uninstall it and then install it again on any other machine. But with Floating user license, you are allowed to do that.
3) Run Time License
The above mentioned two types of licenses are usually bought to “develop” the scripts. So these are “development” Licenses. To execute the scripts on different machines, you need to have the “execution” or “Runtime” license for each machine.
For example, if a tester needs to develop and execute test cases on the same machine, then one development license is enough.
But if he needs to develop on one machine and execute the test cases on three different virtual or physical machines, he needs to buy one “development” license and three runtime licenses.
Some vendors offer free run time licenses (Like Coded UI) and some offer with a price (like Test Complete, Ranorex etc.). So it all depends on vendor to vendor.
4) Open Source License
It is your company’s choice to go for a commercial tool and pay a cost or go for any open source tool.
Commercial tools are expensive, but they offer great support and are easy to use with lots of training material provided. Commercial tools are usually “one tool for all needs”. Open source tools are free, but are generally harder to learn. There is little official support, but you can find solutions by visiting different forums. Open source solutions are normally for specific needs.
b) Support, Upgrade and Training Fee:
For support, training and an upgrade fee, you may need to call the company representative. Some companies offer special discounts on bulk purchase of licenses, so sometimes this information is not clearly mentioned on websites. You will get the information through call or emails only.
Question #3: Does the tool support the Operating System/Browser or Device in which your application runs?
This question is normally dependent on the type of application you are using.
a) If Desktop Based:
If you are working on a desktop application, you should outline that on how many operating systems you want to test that application. I was working on a desktop based application and I wanted to test it on windows 7 and windows 8.1. So I chose Coded UI, because it supports both.
b) If Browser Based
If you are working on a web application, you should outline that on how many browsers you want to test this application. I wanted to execute my test cases on FireFox, Chrome and IE. I chose selenium for my web application because it supports all these browsers. Do make sure that the tool you choose should support both older and newer versions of your required browsers.
c) If Mobile Based
If you are working on Mobile applications, you should know that on which mobile operating systems you have to run your test cases. If your application runs on both android and IOS, your tool should support that. Selenium has separate drivers to run scripts on Android, IOS, Windows Phone and BlackBerry. You can also use a separate tool for each of the Mobile OS. There is Robotium for android, Appium for both IOS and android and CodedUI for Windows phone apps.
Again, this comes to the debate of open source vs commercial. As you can see there are separate open source tools to test web based, mobile based and desktop based applications. But if you go for a commercial tool like Test complete, Ranorex or Test Studio, they can test all three types (Mobile, Desktop and Browser Based Applications). So in case of commercial tool, you need to learn only one tool to test web, desktop and mobile applications.
Question #4: Does the tool support the technologies and third party controls used in your application?
This is a very important aspect while selecting the tool. You should know first hand that what technologies are being used in your application. Consult your developers and write these down. If they are using HTML 5 or SilverLight in web applications, beware, there are not many automation tools to support them. If the tool claims support for these technologies, download the trial version of that tool and try to identify different objects in your application. If the tool fails to identify them, then their claim is false. That activity will save you from the afterwards misery.
The Following table compares different tools with respect to their licensing price and their support for different technologies. (You should take this chart as a learning practice in how to create comparisons between different tools, but the accuracy of the data given is not 100%)
(Click on image to view enlarged)
Y = Supported, N =Not Supported, U = Unknown
Question #5: How many languages the tool supports? Do you have skilled resources for these languages?
Learning the tool is one aspect. Learning the language is another aspect. If you have resources that have expertise in Java, and your tool does not support Java, the time to learn the new language will be added to your automation effort.
Another aspect is that if your product is built on Java, you must have a team of developers that are experts in Java. These developers can also help the automation team in terms of Language related issues. Selecting the tool which offers a language which is familiar to your resources is important and it will help you to minimize the learning curve for your resources.
Question #6: Does the tool support connecting with different Data Sources?
If we are using an automation framework like keyword driven or data driven, we need to have the ability to connect our tool to any data source. If the tool provides connectivity with different data sources easily, it will be very beneficial.
See the support for common data sources such as a .CSV file, Excel File, XML file and Database. If these are present in a tool, you are good to go.
Question #7: How is the reporting mechanism of the automation tool?
When we execute the script, it will either pass or fail. In case of pass, there is not much information needed except for the duration and environment info. But in case of failure, we need a comprehensive report about the failure. The report should tell us that exactly on which step the script fails. A snapshot of the moment of failure will be an added advantage.
Also, this report should be exported to different formats so that we can share this with stakeholders. In many tools, these options are built-in and in some tools there are ways to make your report comprehensive. This is another thing to look out when you download the trial version of the tool. If it is giving comprehensive reports on failures, it’s best for the organization.
Question #8: Can the tool be integrated with test case and bug management repositories?
There is a good chance that your organization is already using any test case or bug management tool. Companies obviously want their automated tool to be integrated with their existing test case management tool so that their whole application life cycle is managed properly. This aspect should also be seen while selecting the test automation tool.
QTP supports QLM, Coded UI supports TFS and TestComplete supports QAComplete. Some Open source tools also have support to integrate with existing open source test management tools. It all depends what your organization is actually using.
Question #9: How is official technical support of the tool?
Here we are talking about commercial tools only. When you select a commercial tool, their support aspect is very important. See the training material provided on the website. Does the website contain videos and tutorials? Does the website has an official forum to ask questions? Download the trial and shoot a question on their forum and see in how many days it gets answered. Do they provide support through call?
The above questions should really be asked every time because you are spending a good amount of money on the tool. If the tool does not have good support, don’t bother to buy it.
Question #10: Some Technical Aspects to see:
There are some other technical aspects to see as well such as:
a) Record and playback support
It is not a recommended approach in test automation, but it is good to have in a tool. It simplifies the learning process of the tool and helps easy scenarios to be easily automated.
b) Different Object Recognition methods and Object Mapping Support
Similarly, there should be an option to properly Map these objects in the object repository. This repository should be easily update-able and managed. Just to remind you that Selenium does not have built-in support for object mapping.
c) Different Checkpoints or Assertion Support.
The test case is passed or failed based on checkpoints or assertions. If the tool has a variety of methods to check your expected results, it is beneficial. QTP has a variety of checkpoints such as Standard, Bitmap, Table, XML, Database and File Content Check Points.
d) Recovery Scenarios Handling.
If the test case fails and you want to continue execution, does the tool support that easily? If recovery scenarios are easy to manage in a tool, it will allow you to execute test cases without any glitch. You can run the test cases at night and in the morning you get the results stating which test cases are failed and which test cases are passed. This will happen only if recovery from failed test cases can be easily managed by the tool. Otherwise, a good amount of automation effort will be wasted in handling recovery scenarios. See recovery scenarios management in QTP.
Always remember that no tool is a good tool or bad tool. It all depends on your requirements and product nature.
Selenium may be the most popular automation tool, but if your product is desktop based, this tool has no use for you. Understand your product first and then search for the appropriate tool which matches your requirement using the guidelines mentioned in this tutorial.
Correct automation tool selection plays a vital role in successful automation.
Next tutorial – Our next tutorial in this series is on ‘Script development and automation frameworks with examples’. Again, check all tutorials in this series on this page.
Feel free to post your queries/comments below about selecting the right automation tool.