How to Develop Test Scripts Using Top 5 Most Popular Test Automation Frameworks (Examples)

When you begin to learn about test automation, you must come across the term “test automation framework”. Maybe some of you get uncomfortable with this term and begin to feel that it is something which is difficult to understand and even more difficult to implement.

This tutorial is written with the aim to help you understand test automation frameworks as simply as possible. Read all tutorials in this Automation Testing Tutorials’ series here.

Test automation framework (in very simple language) is “set of rules.” Rules help us to write scripts in such a manner that result in “lower maintenance”.

To completely understand the concept of framework, we first have to learn how we write simple scripts and then how to implement a framework on them.

test scripts using automation frameworks

In test automation, we write scripts. Scripting is basically about three ‘A’s:

  1. ARRANGEMENT
  2. ACTION
  3. ASSERTION

Below are the details of each A, with examples:

#1. ARRANGEMENT or Object Identification

We identify objects (buttons, dropdowns etc.) either by their ids, names or by their Window Titles etc.

In case of web application, we identify by using ID, or By Xpath or By CSS or By Class Name etc. If nothing works, we then identify objects by using mouse coordinates (But it is not a reliable method of object identification)

Take this example of Selenium WebDriver (with C#) in which we identify objects using the id. (Web application)

 IWebElement txtServer = _webDriver.FindElement(By.Id("tfsServerURL")); 

Another example from MS Coded UI (desktop application)

 WinButton btnAdd = new WinButton(calWindow);
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add"; 

After identification, we arrange or store these objects into UIMaps or Object Repository to reuse them in our scripts. That is why, this step is called ARRANGMENT.

#2. ACTION on the Identified Object

When the objects are identified, we perform some kind of actions on it either by mouse or by keyboard. For example, either we click, or we double click, or we mouse hover over it or sometimes we drag drop. Sometimes we write on text boxes. So any kind of action we perform on these objects are covered in this second step.

Example 1: (Selenium WebDriver with C#)

 txtServer.Clear();
txtServer.SendKeys(“Some sample text”); 

Example 2: (MS Coded UI with C#)

 Mouse.Click(buttonAdd); 

#3. ASSERTION

The assertion is basically checking the object with some expected result. For example, if we press 2+3 on the calculator, the screen should show 5. In this case, our expected result is 5. This concept is already explained in our first tutorial.

Here we give an example of assertion:

 Assert.AreEqual("5", txtResult.DisplayText); 

Nearly every script written in test automation contains these three things: Arrangement, Action and Assertion.

Now take a look at a complete script which contains all these steps. The script will open a calculator, press 1 + 6 and then check whether screen shows 7 or not.

test script for automation framework

Example A:

[TestMethod]
[TestMethod]
public void TestCalculator()
{

var app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");
//Object identification part (ARRANGEMENT)

//----*Calculator Window----*//
WinWindow calWindow = new WinWindow(app);
calWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Calculator";
calWindow.SearchProperties[WinWindow.PropertyNames.ClassName] = "CalcFrame";

//----*Button1 ----*//
WinButton btn1 = new WinButton(calWindow);
btn1.SearchProperties[WinButton.PropertyNames.Name] = "1";

//----*Button Add ----*//
WinButton btnAdd = new WinButton(calWindow);
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add";

//----*Button 6 ----*//
WinButton btn6 = new WinButton(calWindow);
btn6.SearchProperties[WinButton.PropertyNames.Name] = "6";

//----*Button Equals ----*//
WinButton btnEquals = new WinButton(calWindow);
btnEquals.SearchProperties[WinButton.PropertyNames.Name] = "Equals";

//----*Text Box Results----*//
WinText txtResult = new WinText(calWindow);
txtResult.SearchProperties[WinText.PropertyNames.Name] = "Result";

//(ACTIONS Part)
// Click '1' button
Mouse.Click(btn1);

// Click 'Add' button
Mouse.Click(btnAdd);

// Click '6' button
Mouse.Click(btn6);

// Click 'Equals' button
Mouse.Click(btnEquals);

//evaluate the results (ASSERTIONS)
Assert.AreEqual("7", txtResult.DisplayText, “Screen is not displaying 7);

//close the application
app.Close();

}

What’s wrong with that script?

The script is easy to understand and I hope you get the concept of three ‘A’s in the above example. But all is not well with that script.

This script does not allow easy maintenance. Take the example of calculator again, if we have to write test cases of each function of calculator, there will be many test cases. If there are 10 test cases and in each test we have to define the same object, then if any change occurs in the name or id of object, we have to change the object identification part in 10 test cases.

For example, take the example of button ADD in the script.

 WinButton btnAdd = new WinButton(calWindow);
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add"; 

Let’s say, this line is used in 10 test cases. Now in the next version of the calculator, the developer has changed the name of the button from “Add” to “Plus”. Now when we run our test cases, they will be failed and we have to change the above line to this in 10 test cases.

 btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Plus"; 

So we have to improve this test case. We should follow the famous DRY principle in our coding. DRY stands for “Do not Repeat Yourself”. We should write the object identification part in such a manner that the object should be identified only in one place and should be called everywhere.

Take a look at the improved script.

Example B:

//defining the objects outside the script and only once.
ApplicationUnderTest app = null;
public WinWindow calWindow
{

get {
WinWindow _calWindow = new WinWindow(app);
_calWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Calculator";
_calWindow.SearchProperties[WinWindow.PropertyNames.ClassName] = "CalcFrame";
return _calWindow;
}
}

public WinText txtResult
{
get
{
WinText _txtResult = new WinText(calWindow);
_txtResult.SearchProperties[WinText.PropertyNames.Name] = "Result";
return _txtResult;
}
}

//making functions for similar kind of tasks
public void ClickButton(string BtnName)
{
WinButton button = new WinButton(calWindow);
button.SearchProperties[WinButton.PropertyNames.Name] = BtnName ;
Mouse.Click(button);
}

public void AddTwoNumbers(string number1, string number2)
{
ClickButton(number1);
ClickButton("Add");
ClickButton(number2);
ClickButton("Equals");
}

//Test case becomes simple and easy to maintain.
[TestMethod]

public void TestCalculatorModular()
{
app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");

//do all the operations
AddTwoNumbers("6", "1");

//evaluate the results
Assert.AreEqual("7", txtResult.DisplayText, “screen is not displaying 7”);

//close the application
app.Close();

}

In the above example, we have separated the calWindow and txtResult objects and move them to the top so that they can be used across different test methods. We have defined them only once and we can use them in as many test cases as we want.

We have also created two functions. ClickButton() which accepts a button name and click on it and AddTwoNumbers() which takes any two number and add them using the clickButton function inside it.

The moment we start “improving” our code and making it reusable and maintainable, it means, we are making use of any automation framework. Now it gets interesting.

See Also => Why do we need framework for test automation?

There are five popular frameworks in test automation:

  1. Linear
  2. Modularity
  3. Data Driven
  4. Keyword Driven
  5. Hybrid

We will now explain each framework with the help of its characteristics.

#1. Linear Framework:

Characteristics

------------

  • Everything related to script is defined inside the scripts.
  • Does not care about abstraction and code duplication
  • Record and playback normally generate linear code
  • Easy to get started
  • Maintenance Nightmare.

By reading the above 5 characteristics of Linear Framework, we can easily relate our Example A to them. This example is basically using Linear framework, Ever thing related to script is defined inside the script. The calWindow and TxtResult is defined inside the script. The script does not care about abstraction and code duplication. It is also a maintenance Nightmare as I have explained earlier.

So why should we use this framework?

This framework can be used in small scale projects where there are not many UI screens. Also, when we use any automation tool for the first time, it normally generates code in Linear Form. So we can learn about what code is generated by Automation tool for specific actions. Other than these reasons, this framework should be avoided in your scripting.

=> See here example of Linear and Keyword Framework with QTP example.

#2. Modularity Framework:

Characteristics

  • The objects are defined once and reusable in all test methods.
  • Small and to-the-point methods are created for individual functionalities
  • The test case is the collection of these small methods and reusable objects
  • This allows us to write maintainable code.

By Reading the above characteristics, we can relate our Example B to these characteristics. In that example, we have created an abstraction by moving the calWindow to the top and define it inside a property which can be used everywhere. We have created two small and independent functions called ClickButton() and AddTwoNumbers() . We combine these two small functions to create our final script which tests the “Add” functionality of calculator.

This results in easier maintenance. If any change occurs in the calculator UI, we have to change only in functions. Our scripts will remain intact. This framework is highly used in automation. The famous Page Object Framework (which is used with Selenium) is also a kind of modularity framework. We distribute the whole web application into separate pages. The buttons, dropdowns and checkboxes of each page are defined inside the class of that page. If any change occurs on the website, we have to change only in that page class and other pages remain intact. This results in better maintenance and easier readability of scripts.

The only downside of this framework is that, it requires good Object Oriented concepts and strong development skills. If you have those, this framework is highly recommended.

#3. Data Driven Framework:

Characteristics:

  • Test Data (input and output values) are separated from the script and stored in External Files. This could be a .CSV file, an Excel spreadsheet or a Database.
  • When the script is executed, these values are picked from external files, stored in variables and replace the hard coded values if present.
  • Really useful in places where the same test case has to be run with different inputs.

Example C:

We want to run the add test case with three different inputs.

The data is
7 + 2 = 9
5 + 2 = 7
3 + 2 = 5

We stored this data (both input and output) in an external CSV file.

Data driven framework example

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\data.csv", "data#csv", DataAccessMethod.<strong>Sequential</strong>), DeploymentItem("TestCalc\\data.csv"), TestMethod]

public void TestCalculatorDataDrivsen()
{
app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");

//do all the operations
AddTwoNumbers(FromCSV.ADD1, FromCSV.ADD2);

//evaluate the results
Assert.AreEqual(FromCSV.Sum, txtResult.DisplayText);

//close the application
app.Close();

}

In the above script, we define our data source on the top of the script, which is a .csv file.

We have given the path of that .CSV file and tell the script to parse it “Sequentially”. That means the script will run as many times as there are rows present in the csv file. In our case, the script will run 3 times. In each run, it will add the two numbers defined in first two columns and verify that the sum of these two numbers match the number present in the third column.

There are various advantages of this framework. All the values are stored outside the script, so if any change will occur in the next build, we just have to change the data in the external file and the script will remain intact.

The second advantage is that, the same script can be run for different inputs. Take the example of an ERP in which you have to test the registration of 100 employees. You can write one script and store the names and other data related to employees in an external file. You will execute one script and it will run 100 times. Each time with different employee’s data. You can easily detect, on what data the script fails to register the employee. It will be an added advantage when you are doing negative testing.

=> See here example of Data driven and Hybrid framework with QTP example.

#4. Keyword-Driven Framework:

Characteristics:

  • Both data and actions are defined outside the script.
  • It required the development of keywords for different types of actions.
  • The functionality which we have to test is written in a step by step manner in tabular form using the keywords we develop and the test data. We store this table in external files just like data driven framework.
  • The script will parse this table, and perform the corresponding actions.
  • Allows manual tester who does not know about coding to be part of automation to some extent.

Example D:

We defined the data (e.g. 1 + 3 = 4) as well as actions (e.g. Click, Clear etc.) in an excel file in tabular form.

Keyword driven framework example

The script will become something like this (the below code is just written for understanding purpose)

[TestMethod]
public void TestCalculator()
{
app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");

Table tb = ReadFromExcel();
Foreach(WinRow row in tb)
{

WinCell Window = row.Cells[“Window”];
WinCell Control = row.Cells[“Control”];
WinCell Action = row.Cells[“Action”];
WinCell Arguments = row.Cells[“Arguments”];
UITestControl c = GetControl(Control.Text,Window.Text);

If(Action.Text == “Click”)
Mouse.Click (c);

If (Action.Text == “Clear”)
c.Clear();

if(Action.Text == “Verify Result”)
Assert.AreEqual(c.Text, Arguments.Text)

//….and so on
}

} 

The above script is just a parser of the excel file. It parses the excel file line by line and look for keywords to perform respective actions. If it finds the keyword “Click”, it will click on the defined object. If it finds “Verify Result”, it will perform the assertion.

There are various advantages of using the keyword driven framework.

The first advantage is that, this framework is very helpful in those scenarios where there are great chances of changes in test cases. If any step changes in a test case, we don’t need to touch the code. We just have to update the excel file and script will be updated.

You can define all your scripts in an excel file and hand over this excel file to manual testers to add new scripts or update the existing ones. In this way, manual testers can also become the part of test automation because they do not need to code anything. They will just update this excel file when there is a need and scripts will automatically be updated.

The second advantage is that your script becomes tool independent. You can maintain your scripts in an excel file and if you need to change your automation tool at some point, you can easily change it by writing an excel parser in another tool.

The downside of this framework is that, you need to invent keywords for various types of actions. In large scale projects, there will be so many keywords that you need to remember and organize your scripts and keywords. This itself becomes a cumbersome task at a point.

In some complex scenarios, where objects cannot easily be identified and we need to use mouse co-ordinates and other techniques, this framework is not very helpful.

Keyword driven is still a favorite framework for many automation testers. Robot framework by Google is a popular keyword driven framework which is supported by an active community.

#5. Hybrid Test Automation Framework:

Characteristics:

  • The combination of two or more of the above techniques, taking from their strengths and minimizing their weaknesses.
  • The framework can use the modular approach along with either data-driven or keyword-driven framework.
  • The framework can use scripts to perform some tasks that might be too difficult to implement in a pure keyword driven approach.

In simple words, Hybrid framework, use the combination of the above mentioned techniques. We can use data-driven framework which is also modular in nature. For some test cases, we can use keyword-driven approach and for remaining we can use modular. So whenever we mix two or more techniques mentioned in this article, we are actually using a Hybrid approach.

Conclusion

I hope that test automation framework is no longer a scary term for you now. I tried to explain the most popular frameworks as simply as possible.

Frameworks are here to make your life easier. They help you to write maintainable and reliable scripts. Without using frameworks, the test automation field is a nightmare. For every small change in application, you have to change your code in hundreds of places.

So understanding of these frameworks is a must for every tester who wants a taste of test automation.

In our next tutorial in this series we will learn ‘Execution and reporting of Test Automation’.

If I have missed anything in this article or you need to ask any question, please feel free to ask in the comments section.

Recommended reading

23 comments ↓

#1 Kameshwar

I do not C# , Why you do not use java, as previous tutorial. Please use java, when you give any example.

#2 Kameshwar

I do not know C# , Why you do not use java, as previous tutorial. Please use java, when you give any example.

#3 Mamatha

Hi,

Very useful article for beginners.

I working as a manual tester from last 7 years, i wish to learn and implement automation for my project. When i try to learn tools, i able understand tools(QTP/selenium) theoretically, unable to practice lack of programming/coding skills. Is learning programming first before automation is mandatory? If yes which language is better to learn? Suggest me How to become Automation Tester…

Thanks for your time and consideration.

#4 Rajesh Sampath

Very good start and well explained.

#5 Dilip

A very good article to have a brief idea about Automation Frameworks.I would like to know about how to handle the app crash scenarios during the Test Execution.

#6 Tejaswini

Very nice description
But basic problem is, scripts are not readable in one shot.it is really inconvenient to scroll and come again to the starting line to read exact script .

#7 Guru

Very good explanation.
Please do post some more generic examples, which will be helpful for beginners like me.

#8 Vijay

@ Tejaswini
Now you can read scripts in one shot. Disabled scrolling and enabled line wrapping.

#9 Mohammad Saad

@Kameshwar: I have used C# in all my previous tutorial as well. You can use this tool
https://www.varycode.com/converter.html
to convert my c# examples into java for your understanding.

#10 Mohammad Saad

@Mamatha: Thanks for the feedback. Yes you need to be good in programming if you want to become an automation engineer. please read this article
http://www.softwaretestinghelp.com/automation-testing-tutorial-2/ in which I have described this in detail.
For language, I suggest to learn either Java or C# as they both are very powerful languages and lot of tools support these two.

#11 Mohammad Saad

@Rajesh Sampath: Thank you

#12 Mohammad Saad

@Dilip : Thank you for your feedback. Recovery Scenarios will be discussed in Article number 6: Execution and reporting in test automation

#13 Mohammad Saad

@Tejaswini: Thanks for your feedback. I saw that vijay has fixed your issue

#14 Mohammad Saad

@Guru: Thank you for your feedback. I have given the example of Calculator which is the most simplest example anybody can understand. For more examples, you can browser other articles present in this website. Check out Selenium and QTP tutorials, there you can find many examples.

#15 Tejaswini

Hey…Thank you sir…:-)
Scripts are now readable .
Now waiting for next ‘Execution and reporting of Test Automation’

#16 Veru

thanks for the post

#17 Shivaprasad

Hi, this is really good stuff. For a beginner like it will make the things clear upfront. Thanks

#18 Shivaprasad

Hi, this is really good stuff. For a beginner like me it will make the things clear upfront. Thanks for this article.
I am into testing and I am learning selenium, any suggestions…

#19 mahalakshmi

Plz give more information on testing tool selenium

#20 Ajay

Good stuff..got some ideas to cross questioning…

#21 sandip mahajan

I m in last year of my graduation in computer science….!
I m little bit weak in programing….!
Should i choose testing as a career option ahead….!
Please help…..!

#22 Karthik Rajesh

Nice article

#23 subodh kukreti

Very good and clear article to explain the scary term framework. And well said by author that now after reading this framework isn’t be scary word for you.
thanks

Leave a Comment