What is Automation Testing (Ultimate Guide to Start Test Automation)

A Complete Guide to start Automation Testing on your project: 

What is Automation Testing?

Automation testing is a Software testing technique to test and compare the actual outcome with the expected outcome. This can be achieved by writing test scripts or using any automation testing tool. Test automation is used to automate repetitive tasks and other testing tasks which are difficult to perform manually. 

Do you want to start the automation test on your project but are you struggling with the most basic steps as mentioned below:

  • How to introduce automation to your project?
  • How to select the best and right automation tool?
  • How to develop scripts effectively?
  • How to execute and maintain test scripts?
  • And finally what are the best practices that you need to follow for successful automation testing?

Today, we have planned to enrich your knowledge with a series of tutorials on “Getting started with Automation Testing”. This series of automation tutorials will answer all the above questions in a step by step manner with simple examples.

Let’s take a look at the series of Tutorials on Starting Automation on Your Project!!

Automation End-to-end Process:

Tutorial #1: Best Guide to Start Automation on Your Project
Tutorial #2: Types of Automated Tests and Some Misconceptions
Tutorial #3: 10 Steps to Introduce Automation on Your Project
Tutorial #4: The A to Z Guide on Selecting the Best Automation Tool
Tutorial #5: Script Development and Automation Frameworks
Tutorial #6: Execution and reporting of Automation
Tutorial #7: Best Practices and Strategies of Test Automation

Automation Tips:

Tutorial #8: 10 Tips you should read before automating your testing work
Tutorial #9: How Does Test Planning Differ for Manual and Automation Projects
Tutorial #10: When To Opt For Automation?
Tutorial #11: Automation Test Challenges
Tutorial #12: Guide to Implement Proof of Concept (POC) in Automation
Tutorial #13: How to Select Correct Test Cases for Automation
Tutorial #14: How to Translate Manual Test Cases into Automation Scripts

Automation Career:

Tutorial #15: Tips to Become a Better Automation Tester
Tutorial #16: Test Automation – Is it a Specialized Career? Can Normal Testers Do Automation Also?

Popular Automation Tools:

Tutorial #17: Selenium Tutorials 31+ Best Free Selenium Training Tutorials
Tutorial #18: QTP Tutorials
Tutorial #19: SoapUI Web Services Testing Tool
Tutorial #20: HP LoadRunner for Performance Testing

Automation Frameworks:

Tutorial #21: Why Do We Need Framework for Automation
Tutorial #22: Most Popular Automation Frameworks

Automation in Agile:

Tutorial #23: How to Implement Efficient Automation in the Agile World

Other Automation Tools:

Tutorial #24: Best Automation Test Tools
Tutorial #25: Sikuli GUI Automation Tool
Tutorial #26: PowerShell: Desktop Application UI Automation With PowerShell
Tutorial #27: Katalon Automation Recorder (Selenium IDE Alternative)
Tutorial #28: Geb Tool: Browser Automation Using Geb Tool
Tutorial #29: AutoIt: How to Handle Windows Pop-up Using AutoIt
Tutorial #30: Cucumber: Automation Using Cucumber Tool and Selenium
Tutorial #31: Protractor Testing Tool for End-to-end Testing of AngularJS Applications

Mobile Automation Testing:

Tutorial #32: Appium Studio Hands-on Tutorial
Tutorial #33: Appium Tutorial for Beginners
Tutorial #34: Selendroid Tutorial: Android Mobile Automation Framework
Tutorial #35: Ranorex Tutorial: A Powerful Desktop, Web, and Mobile Testing Tool

Domain Specific Automation Examples:

Tutorial #36: Automation of JAVA/J2EE Applications

Interview Preparation for Automation Jobs:

Tutorial #37: Automation Testing Interview Questions
Tutorial #38: Selenium Interview Questions

***********************************************************************

Let’s explore the first tutorial from “The Ultimate Guide to Automation Testing” series!!

What is Automation Testing?

If a software can do anything then, why can’t a software test a software?

Does this statement sound logical to you?

If yes, then congratulations, you are now thinking about Test Automation, which is the center point that we are going to discuss in this series of informative tutorials.

automation testing guide

Imagine yourself n the first day at your job as an SQA. You are presented with an application to be tested. It’s an ERP application containing 100s of forms and thousands of reports. You begin your exploratory testing by opening a form which contains around 50 different fields.

You try to enter random data in this form which took around 20 minutes. Then you press submit. Wolla!! An error message is shown which looks like an unhandled exception. You become very happy. You proudly note down the steps and report the bug in your bug management system. Great effort, you feel really confident and energetic. You continue the testing until the day ends and find some more bugs. “Amazing first day”, you thought.

Now comes the next day, the developer has fixed the issue and releases a new version of the build. You test the same form with the same steps and you found that the bug is fixed. You mark it fixed. Great effort. You have contributed to the quality of the product by identifying that bug and as this bug is fixed, the quality is improved.

Now comes the third day, a developer has again released a newer version. Now you again have to test that form to make sure that no regression issue is found. Same 20 minutes. Now you feel a little bored.

Now imagine 1 month from now on, newer versions are constantly releasing and on every release, you have to test this lengthy form plus 100 of other forms like this, just to make sure that no regression is there.

Now you feel angry. You feel tired

. You begin to skip steps. You fill around only 50% of the total fields. Your accuracy is not the same, your energy is not the same and definitely, your steps are not the same.

And one day, the client reports the same bug in the same form. You feel pathetic. You feel unconfident now. You think you are not competent enough. Managers are questioning your ability.

I have a news for you; this is the story of 90% of the manual testers out there. You are not different.

Regression issues are the most painful issues. We are humans. And we cannot do the same thing with the same energy, speed and accuracy every day. This is what machines do. This is what automation is required for, in order to repeat the same steps with the same speed, accuracy and energy as they were repeated the first time.

I hope you get my point!!

automation testing a friend

Whenever such a situation arises, you should automate your test case. Test automation is your friend. It will help you to focus on new functionality while taking care of the regressions. With automation, you can fill that form in less than 3 minutes.

The script will fill all the fields and tell you the result along with screenshots. In case of failure, it can pinpoint the location where the test case failed, thus helping you to reproduce it with ease.

Scenarios which require Automation

The above scenario is not the only case when you will need automation testing. There are several situations, which cannot be tested manually.

For Example,

  1. Comparing two images pixel by pixel.
  2. Comparing two spreadsheets containing thousands of rows and columns.
  3. Testing an application under the load of 100,000 users.
  4. Performance Benchmarks.
  5. Testing the application on different browsers and on different operating systems in parallel.

These situations require and should be, tested by tools.

So, when to automate?

This is an era of agile methodology in SDLC, where the development and testing will go almost in parallel and it is very difficult to decide when to automate.

Consider the following situations before stepping into automation

  • The product may be in its primitive stages, when the product does not even have a UI, at these stages we must have a clear thought on what we want to automate. The following points should be remembered.
    • Tests should not be obsolete.
    • As the product evolves it should be easy to pick on the scripts and add on to it.
    • It is very important to not get carried away and ensure that the scripts are easy to debug.
  • Do not attempt UI automation at the very initial stages as UI is subjected to frequent changes, thereby will lead to scripts failing. As far as possible opt for API level/Non UI level automation until the product stabilizes. API automation is easy to fix and debug.

How to Decide Best Automation Cases:

Automation is an integral part of a testing cycle and it is very important to decide what we want to achieve with automation before we decide to automate.

The benefits that automation seem to provide are very attractive, but at the same time, an ill-organized automation suite can spoil the entire game. Testers may end up debugging and fixing the scripts most of the time resulting in loss of testing time.

Efficient Test Automation

This series explains you about how an automation suite can be made efficient enough to pick up the right tests cases and yield the right results with the automation scripts that we have.

Also, I have covered the answers to questions like When to automate,  What to automate, What not to automate and How to strategize automation.

Right Tests for Automation

The best way to tackle this problem is to quickly come up with an “Automation Strategy” that suits our product.

The idea is to group the test cases so that each group will give us a different kind of result. The Illustration given below shows how we could group our similar test cases, depending on the product/solution that we are testing.

Grouping Test Cases

Let us now dive deep and understand what each group can help us to achieve:

#1) Make a test suite of all the basic functionality Positive tests. This suite should be automated, and when this suite is run against any build, results are shown immediately. Any script failing in this suite leads to S1 or S2 defect, and that build specific can be disqualified. So we have saved a lot of time here.

As an additional step, we can add this automated test suite as a part of BVT (Build verification tests) and check the QA automation scripts into the product building process. So when the build is ready testers can check for the automation test results, and decide if the build is suitable or not for installation and further testing process.

This clearly achieves the goals of automation which are:

  • Reduce testing effort.
  • Find Bugs at earlier stages.

#2) Next, we have a group of End to End tests.

Under large solutions, testing an end to end functionality holds the key, especially during the critical stages of the project. We should have a few automation scripts that touch upon the end to end solution tests as well. When this suite is run, the result should indicate whether the product as a whole is working as it is expected or not.

The Automation test suite should be indicated if any of the integration pieces are broken. This suite need not cover each and every small feature/functionality of the solution but it should cover the working of the product as a whole. Whenever we have an alpha or a beta or any other intermediate releases, then such scripts come in handy and give some level of confidence to the customer.

To understand better let’s assume that we are testing an online shopping portal, as part of the end to end tests we should be covering only the key steps involved.

As Given Below:

  • User login.
  • Browse and select items.
  • Payment Option – this covers the front end tests.
  • Backend order management (involves communicating with multiple integrated partners, checking stock, emailing the user etc) – this will help the testing integration of individual pieces and also the crux of product.

So when one such script is run it gives a confidence that the solution as a whole is working fine.!

#3) The third set is the Feature/Functionality based tests.

For example, We may have a functionality to browse and select a file, so when we automate this we can automate cases to include the selection of different types of files, sizes of files etc, so that feature testing is done. When there are any changes/additions to that functionality this suite can serve as a Regression suite.

#4) Next on the list would be UI based tests. We can have another suite that will test purely UI based functionalities like pagination, text box character limitation, calendar button, drop downs, graphs, images and many such UI only centric features. Failure of these scripts is usually not very critical unless the UI is completely down or certain pages are not appearing as expected!

#5) We can have yet another set of tests that are simple but very laborious to be carried out manually. Tedious but simple tests are the ideal automation candidates, for example entering details of 1000 customers into the database has a simple functionality but extremely tedious to be carried out manually, such tests should be automated. If not, they mostly end up getting ignored and not tested.

What NOT to Automate?

Given below are few tests which should not be automated.

#1) Negative tests/Failover tests

We should not attempt on automating negative or failover tests, as for these tests the testers need to think analytically and negative tests are not really straightforward to give a pass or fail result which can help us.

Negative tests will need a lot of manual intervention to simulate an actual disaster recovery kind of scenario. Just to exemplify we are testing features like web services reliability – to generalize it here the main aim of such tests would be to cause deliberate failures and see how well the product manages to be reliable.

Simulating the above failures are not straightforward, it can involve injecting some stubs or use some tools in between and automation is not the best way to go here.

#2) Ad hoc tests

These tests may not really be relevant to a product at all times and this may even be something that the tester could think of at that stage of project initiation, and also the effort to automate an ad-hoc test has to be validated against the criticality of the feature that the tests touch upon.

For example, A tester who is testing a feature which deals with compression/encryption of data might have done intense ad-hoc tests with the variety of data, file types, file sizes, corrupt data, a combination of data, using different algorithms, across several platforms etc.

When we plan for an automation we may want to prioritize and not do an exhaustive automation of all the ad hoc tests for that feature alone, and end up with a little time for automating the other key features.

#3) Tests with massive pre-setup

There are tests the that require some enormous pre-requisites.

For example, We may have a product that integrates with a 3rd party software for some of the functions, like product integrates with any messaging queue system which requires installation on a system, setting up of queues, creating queues etc.

The 3rd party software could be anything and the setup may be complex in nature and if such scripts are automated then these will forever be dependent on the function/setup of that 3rd party software.

Pre-requisite include:

At present things may look simple and clean as both side setups are being done and all is fine. We have seen on numerous occasions that when a project enters the maintenance phase the project is moved to another team, and they end up debugging such scripts where the actual test is very simple but the script fails due to a 3rd party software problem.

The above is just an example, in general, keep an eye on tests that have laborious pre setups for a simple test that follows.

Simple Example of Test Automation

When you are testing a software (on the web or desktop), you normally use a mouse and keyboard to perform your steps. Automation tool mimics those same steps by using a scripting or a programming language.

For Example, if you are testing a calculator and the test case is that you have to add two numbers and see the result. The script will perform the same steps by making use of your mouse and keyboard.

The example is shown below.

Manual Test Case Steps:

  1. Launch Calculator
  2. Press 2
  3. Press +
  4. Press 3
  5. Press =
  6. The screen should display 5.
  7. Close Calculator.

Automation Script:

//the example is written in MS Coded UI using c# language.
[TestMethod]
public void TestCalculator()
{

//launch the application
var app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");

//do all the operations
Mouse.Click(button2);
Mouse.Click(buttonAdd);
Mouse.Click(button3);
Mouse.Click(buttonEqual);

//evaluate the results
Assert.AreEqual("5", txtResult.DisplayText,”Calculator is not showing 5);

//close the application
app.Close();
}

The above script is just a duplication of your manual steps. The script is easy to create and easy to understand as well.

What are Assertions?

The second last line of the script needs some more explanation.

Assert.AreEqual(“5”, txtResult.DisplayText,”Calculator is not showing 5);

In every test case, we have some expected or predicted result at the end. In the above script, we have an expectation that “5” should be shown on the screen. The actual outcome is the result that is displayed on the screen. In every test case, we compare the expected outcome with the actual outcome.

Same goes for automation testing as well. The only difference here is, when we do that comparison in test automation, then it is called something else in every tool.

Some tools call it as “Assertion”, some call it as “checkpoint” and some call it as “validation”. But basically, this is just a comparison. If this comparison fails, for E.g. a screen is showing 15 instead of 5 then this assertion/checkpoint/validation fails and your test case is marked as failed.

When a test case is failing due to an assertion then that means you have detected a bug through test automation. You must report it to your bug management system just like you normally do in manual testing.

In the above script, we have performed an assertion in the second last line. 5 is the expected outcome, txtResult. DisplayText is the actual outcome and if they are not equal, we will be shown a message that “Calculator is not showing 5”.

Automation – A Cost-effective Method for Regression Testing

Automation costs are really higher initially. It includes the cost of the tool, then the cost of the automation testing resource and his/her training.

But when the scripts are ready, they can be executed hundreds of times repeatedly with the same accuracy and rather quickly. This will save many hours of manual testing. So the cost gradually decreases, and ultimately it becomes a cost-effective method for Regression testing.

Conclusion

Often testers come across project deadlines and mandates to automate all the cases to improve testing estimates.

There are some common “wrong” perceptions about automation.

They are:

  • We can automate every test case.
  • Automating tests will reduce testing time enormously.
  • No bugs are introduced if automation scripts are running smoothly.

We should be clear that automation can reduce testing time only for certain types of tests. Automating all the tests without any plan or sequence will lead to massive scripts which are heavy maintenance, fail often and need a lot of manual intervention too. Also, in constantly evolving products automation scripts may go obsolete and need some constant checks.

Grouping and automating the right candidates will save a whole lot of time and give all the benefits of automation.

This excellent tutorial can be summarized in just 7 points.

Automation Testing:

  • Is the testing which is done programmatically.
  • Uses the tool to control the execution of tests.
  • Compares expected outcomes with the actual outcomes (Assertions).
  • Can automate some repetitive but necessary tasks (E.g. Your regression test cases).
  • Can automate some tasks which are difficult to do manually (E.g. Load testing scenarios).
  • Scripts can run quickly and repeatedly.
  • Is cost effective in the long run.

Here, Automation is explained in simple terms, but that doesn’t mean that it is always simple to do. There are challenges, risks and many other obstacles involved in it. There are numerous ways by which test automation can go wrong, but if all goes well, then the benefits of test automation are really huge.

Upcoming ones in this series:

In our upcoming tutorials, we will discuss several aspects related to automation.

These include:

  1. Types of automated tests and some Misconceptions.
  2. How to introduce automation in your organization and avoiding common pitfalls when doing test automation.
  3. The tool selection process and comparison of various automation tools.
  4. Script Development and Automation Frameworks with examples.
  5. Execution and reporting of Test Automation.
  6. Best Practices and Strategies of Test Automation.

Are you eager to know more about each and every concept of Automation Testing? Watch out and stay tuned to our list of upcoming tutorials in this series and feel free to express your thoughts in the comments section below.

NEXT Tutorial#2


51 thoughts on “What is Automation Testing (Ultimate Guide to Start Test Automation)”

  1. Hi,

    Article was really helpful.Looking forward to read more articles on “Guide to Start Automation Testing on Your Project”…That would be very helpful to switch to automation.

  2. Nice Article. I need guidance on writing test scenarios in Behat. Can you help on this?

    Share any article with Behat, Mink & Selenium. How to use it with the project with examples?

  3. @komal: Thank you. For tool selection process, please wait for 4th Article in this series. The article name is “The tool selection process and comparison of various automation tools.”

  4. very good article specially for new comers who want to make their career in test automation field..i need a complete detail article about test automation in automotive industries.which tools are used and mostly which targets are needed test automation.thanks.

  5. Indeed a good article, thanks for the effort you put to consolidate those basic steps together.

    It would be better if you add the tool name in the subject line which code is given as example (Selenium/QTP/RFT)

    The topic is not elaborate enough to justify the title. I mean guide to start automation should include a bit of manual test case , considering and filtering those for regression suite and then identify keywords, then prepare test data , then record and play/ script code then enhance those with programming logic and write reporting module , then outputs, then maintaince and debug … something like this. But maximum text covers why we need automation, advantages of automation etc.. Just a thought for improvement no other intention!.Correct me if I am wrong.

  6. I have started Test Automation using Microsoft Test Manager, MS Coded UI Test, and Visual Studio Tools. I am Looking forward to read more articles on the same.

  7. Pramod Mallick: Thank you for your feedback. This is just the first article in the series of 7 articles about test automation. Your questions will be answered in upcoming Articles.

  8. Pramod Mallick: Please see the comments on the top of automation script. The tool name is mentioned there. (MS Coded UI)

  9. I need little more information on working of Selenium. How to use it with the project with examples?Some good sources to get some hands on this tool(Sites information, Prerequisite for the same)

  10. Mohammad Saad:
    Thanks for your comments. I am eager to read next part of the article.
    If possible, cover the differences between coded UI and other tools.
    Example: QTP vs Coded UI – including technology, simplicity, cost, versioning, maintaing, compatibility, depedency etc..

    Thanks,
    Pramod Mallick

  11. Thank you very much for this tutorial and initiative. I’ve started learning Automation Testing with OpenSource Selenium2 (WebDriver) requiring Java programming knowledge and now I’m trying to get familiar with UFT ver: 12.02 from HP (Unified Functional Testing) former-QTP that requires now knowledge of Visual Basic scripting. I’ve read all the comments above and now my questions are:
    1. Are MS Coded UI and Microsoft Test Manager related to UFT or are they totally different and require to learn other scripting languages?
    2.Which one’s better for WebApp and Mobile Testing?

    Thank you,
    Luke

  12. @Luke: Thank you for your feedback. The answer to your first question is that MS Coded UI and MS Test Manager are totally different tools. These are created by Microsoft. UFT on the other hand is owned by HP. You need to learn either C# or VB.Net to learn Coded UI. MS Test manager is a test case and bug management tool so you don’t need to learn any programming language for that.

    The answer to your second question is described in detail in article number 4 “The tool selection process and comparison of various tools” which will be published shortly.

  13. @Pramod Mallicki These differences are described in detail in article number 4 “The tool selection process and comparison of various tools” which will be published shortly.

  14. One more thing it is fine as you said we need to introduce automation for project/product which we required more regression testing but I need to know about if project is small size and not much regression testing work then automation is feasible or not ? If yes then why or not then why ?

  15. @Keyur Makwana: Your question will be answered in article number 2 and article number 3 in this series. You will get to know that when you will need automation and when you can avoid automation all together.

  16. @Mohammad Saad Thank you for your prompt and clear answer. I’m looking forward for your upcoming articles.

    Thank You,
    Luke

  17. Thanks for this basic intro to automation, Mohammed. I look forward to your other articles.

    I hope that our wiki on test automation will also be useful (TestAutomationPatterns.wikispaces.com).

    I will also be interested to see if you reference my 2 books on test automation! ;-)

  18. Great start!!! You explained automation concept so simply and precisely that I got it in first read. Please publish the remaining articles soon. I can hardly wait to delve deeper into this. I am itching to introduce automation at my work. I know a little bit about Selenium but there are so many other options as well that I know nothing about. I look forward to learning more about all the other tools.

  19. Hitesh: Thank you for the compliments. You will get an idea about different tools in article number 3 . “The tool selection process and comparison of various automation tools.” . Keep reading.

  20. Very comprehensive article. Would be very much interested in having some tutorials in MTM. Some comprehensive information about Microsoft Test Manager would be great.

  21. OCCYGEN is a test management tool. In a more general way, it is a tool for QA / Validation activities in a company. Occygen enhances the productivity of testers and improve visibility on the test run progress. It is a project management tool and a rich internet application that can be accessed through any web browser, no installation required. On server side, it can be deployed under Windows or Linux machine. It offers customization of the fields and test case execution using Nosetests based on Python Scripts. Can be integrated with ticket management softwares like Mantis, Bugzilla and Jira.

  22. Good reading your article…I am looking for more information on how to start writing automation.How to develop the test framework

  23. First i would to say many thanks to Mohammad Saad for great information on Automation Testing. i cant wait to read for next Articles . Best of luck.

  24. Hi,
    Thanks for all these tutorial series.

    I am about to start Automation testing in my organization. This tutorials are really big help.

    Looking forward to get good knowldge on selenium.

  25. Awesome article!.. Am new to automation.Could you help on tool selection(open source) for the application which is functionality oriented.

  26. Saad i need to contact you, want to discuss something about how to initiate automation testing from the scratch for a particular product. Need guidance. ThankYou.

  27. Its really informative article explained in very simple language with suitable examples, waiting for next article in the series.

Leave a Comment