I am not a big fan of labels. Here is what I mean by that.
If I have to check few aspects before I determine whether or not QA can be started, I will simply make a list and perform the action. In my opinion, it does not matter if I officially call it a “Test readiness review” operation or not – as long as I am doing what I am supposed to do, I think there is no need to call it a specific name or label.
But I stand corrected. Recently, in my class, I was teaching Agile-scrum model for software development. There was a question ‘how is testing performed in an Agile method?” I was explaining two methods- one is where we try to include it within each sprint and the other is a best practice that I have learnt from the first-hand implementation- which is to lag a QA sprint with respect to the development one.
One of my students asked me if there is a name for the second one and I didn’t because I never placed emphasis on the names themselves.
But at that moment, I felt how important it was to label a process appropriately to make sure that we have a term to refer to the process we are talking about.
Therefore, today we are going to do just that: Learn the process behind the term “Test Harness”.
As I mentioned before in some of my previous articles: a lot can be understood from the literal meaning of the name. So, check your dictionary for what “Harness” means and the big reveal of whether or not, it applies, in this case, is something we will see at the end.
There are two contexts to where Test harness is used:
- Automation testing
- Integration Testing
Let’s begins with the first one:
What You Will Learn:
Context #1: Test Harness in Test Automation
In the automation testing world, Test harness refers to the framework and the software systems that contain the test scripts, parameters necessary (in other words, data) to run these scripts, gather test results, compare them (if necessary) and monitor the results.
I am going to try and make this simpler with the help of an example.
If I was talking about a project that uses HP Quick Test Professional (now UFT) for functional testing, HP ALM is linked to organize and manage all the scripts, runs and results and the data is picked from an MS Access DB – The following would be the test harness for this project:
- The QTP (UFT) software itself
- The scripts and the physical location where they are stored
- The Test sets
- MS Access DB to supply parameters, data or the different conditions that are to be supplied to the test scripts
- HP ALM
- The test results and the comparative monitoring attributes
As you can see, software systems (automation, test management, etc.), data, conditions, results – all of them become an integral part of the Test harness – the only exclusion being the AUT itself.
Context #2: Test Harness in Integration Testing
Now it is time to explore what Test harness means in the context of “Integration Testing”.
Integration testing is to put together two or modules (or units) of code that interact with each other and to check whether or not the combined behaviour is as expected or not.
Ideally, Integration testing of two modules should and would be possible to carry out when both of them are 100% ready, unit tested and good to go.
However, we do not live in a perfect world- which means, one or more modules/units of code that are to be the constituent elements of the integration test might not be available. To solve this situation we have stubs and drivers.
Stud is usually a piece of code that is limited in its function and will substitute or proxy for the actual module of code that needs to take its place.
Example: To further explain this, let me use a scenario
If there is a unit A and Unit B that are to be integrated. Also, that Unit A sends data to Unit B or in other words, Unit A calls Unit B.
Unit A if 100% available and unit B is not, then the developer can write a piece of code that is limited in its capability ( what this means is the Unit B if it has 10 features, only 2 or 3 that are important for integration with A) will be developed and is used for integration. This is called a STUB.
The integration would now be: Unit A->Stub (substituting for B)
On the other hand, if Unit A is 0% available and Unit B is 100% available, the simulation or proxy has to be Unit A here. Therefore when a calling function is replaced by an auxiliary code, then it is called the DRIVER.
The integration, in this case, would be: DRIVER (substituting for A) -> Unit B
The entire framework: The process of planning, creating and usage of stubs and/or drivers to carry out the integration testing is called the Test Harness.
Note: the above example is limited and the real-time scenario might not be as simple or as straightforward as this. Real-time applications have complex and composite integration points.
As always, STH believes that the even the most technical of definitions can be derived from the term’s simple, literal meaning.
The dictionary on my smartphone tells me that a “Harness” is (look under the verb context):
“To bring under conditions for effective use; gain control over for a particular end; “
Following this and adapting this to testing:
“A test harness simply is to create the correct framework and use it (and all of its constituent elements) to control the entire activity as to get the most of the situation- whether automation or integration. “
There, we rest our case.
A few more things before we finish:
Q. What are the benefits of a Test Harness?
Now, would you ask what the importance of breath for human life is – it is intrinsic, isn’t it? Similarly, a framework to test effectively is like a given. The benefit, if we have to spell it in so many words- I would say, every testing process has a test harness whether we consciously say that it is “The Test harness” or not. It is like travelling knowing the route, the destination and all the other dynamics of the journey.
Q. What is the difference between test harness and test framework?
I personally think that comparing and contrasting is not often the right approach when understanding related concepts because the lines are often blurry. As an answer to that question, I would say, the Test harness is specific and Test framework is generic. For example, a test harness will include the exact information of the test management tool down to the login IDs to be used. A test framework, on the other hand, will simply say that a test management tool will do the respective activities.
Q. Are there any Test Harness tools?
Test harness includes tools – like automation software, test management software, etc. However, there are no specific tools to implement a test harness. All or any tools can be a part of Test Harness: QTP, JUnit, HP ALM- all of them can be constituent tools of any Test Harness.
About the author: This article is written by STH team member Swati S.
And, always with definitions, there are always differences in opinions. We welcome your opinions and love to hear what you think. Please feel free to leave a comment, questions or suggestion below.
14 thoughts on “What is Test Harness and How is it Applicable to us, Testers”
Hi, Interesting article. But again, all these test harness activities are part of Test Planning too. There are many more activities with many more names. But if I see them practically, its all about turning to word called “Planning”. Some activity will go in deep where as some remain on edge.
here is another Wiki definition
In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.
Very good artical actually all the terms are interrealted to each other still good abrivation ‘Test Harness’.????
Test harness explained in the context of Test automation and Integration testing is really good. Naming the term Test Harness sounds cool
Using the same formula as suggested here; when I look up the term harness then I come up with a slightly different understanding and would like to share and validate the same here.
Harness is what mountain climbers, para-gliders etc wear to hold them, it is a combination on strips that hold your torso; head,hands and legs are are out of it.
Taking this definition and comparing it with framework is when a body/item is enclosed completely and has to go through the framework for its interactions to outside world.
So in case of a Test Harness, the automation script being designed uses the set of wrapper classes/functions written but also has access to AUT other wise eg to perform a UI operation you go via selenium (eg) but for a AUT’s DB access you make a direct JDBC connection or in case of a web service TC you take a different route; Hence, Even though called ‘test/automation frameworks’ most of the times they are actually ‘test/automation harness’.
In my opinion, if there is any other import in the script file except for the framework then it is an harness not a framework.
Same thing also applies in case of integration testing or any other automated testing like Performance Testing, etc.
Please share your views and feel free to correct me.
The title of this article should be “What is a Test Harness”
There are some typos in above article:
“Stud is usually a piece of code that is limited in its function and will substitute or proxy for the actual module of code that needs to take its place.”
There should be stub in the starting.
“Q. What is the difference between test harness and test framework?
I personally think that comparing and contrasting in (SHOULD BE is)not often the right approach when understanding related concepts because the lines are often blurry. “
In short, test harnessing is all about test planning. That’s how I read it.
To me, this article explains it all and clearly.
One comment – test harness is a concept term which can be applied by using tools or scripts. I would suggest not use “Framework” to define it.
Very Well Explained..thanks
I had heard that a test framework includes: test execution engine, test case repository, test data and test reporting. Would it be fair to examine the “test harness” (as I’ve used it) as including the test framework, as well as tools and scripts necessary for Version Control (e.g. SVN/CVS, Git), Build (e.g. Ant, Maven, Gradle), Orchestration [whether timed, on-event or on-demand] (e.g. Jenkins, Kubernetes) and test Infrastructure Management (e.g. Docker, Kubernetes, AWS, etc.)? A test framework alone (by the above definition) is rather limited in its ability to support a dynamic software development team, encompassing that test framework within the rest of the test harness extends the reach of the framework to be initiated immediately upon code check-in/push for unit testing, or timed orchestration to test integrated code changes applied across multiple time zones, as well as distributed regression testing or internationalization testing across multiple platforms/browsers in parallel.
Your blogs are rich and thank you for your dedicated work to help thousands, if not millions, of people.
what is a test harness when you are doing manual testing?