Comparison testing is an oft-repeated phrase and a type of testing that evokes our attention. Let’s get into the details of how the comparison test is performed and what it actually means in real time.
What is Comparison Testing?
Comparison testing is all about assessing the strengths and weaknesses of a software product concerning other software products existing in the market. The goal of comparison testing is to provide pivotal and critical information to the business to unravel the software product’s competitive advantage in the market Vis-a-vis loopholes.
What kind of comparison we do depends on the object of testing. For example, the object of testing could be anything like:
- A web application
- ERP application
- CRM application
- A module of an application requiring validation of data after completion of a transaction and so on
Table of Contents:
Establishing Criteria for Comparison Testing
Establishing criteria for comparison tests for a particular software product is a subjective matter determined by the type of software application being tested and use cases specific to the business. The test scenarios we develop are dependent on the type of application and business-specific use cases.
Testing efforts and procedures are always organized in such a way that wherever there is ambiguity, a definite strategy is evolved that can be applied across all projects.
We would, therefore, distribute this testing into two distinct phases
Phases
This testing can be performed in two distinct phases:
- Comparing software products against known standards or benchmarks
- Comparing software products with specific features of other existing software products
a) For Example, if a Siebel CRM application is being tested, we know that any CRM application has modules that broadly deal with capturing customer details, processing customer orders, managing customer requests, and customer issues.
In the first phase of testing, we can test the functionality of the application against known standards and functionality as existing in the market at the time of testing.
We can ask questions like:
- Does the application have all the modules which a CRM application should have?
- Do the modules perform basic functionality as expected?
We will evolve test scenarios in such a way that the test results validate the functionality of the application as against already known standards in the market.
b) In the second phase of testing, we can compare the features of an application against the features of other software products in the market.
For Example, the following features can be considered for comparison with other software products.
#1) Price
#2) Performance of the application
Example: Response time, network load
#3) User Interface (look and feel, ease of use)
In both phases of testing, testing efforts are structured in such a way that the potential areas that can cause disruption to business are identified. An appropriate testing strategy is evolved to direct test design and test execution.
Thorough knowledge of business use cases and requirements is inevitable.
The Structured Way of Performing Comparison Test
Examples of Test Scenarios for a CRM Application
Let us take the example of a CRM application for the purchase of a mobile for the purpose of test scenarios.
We know that any such CRM application should broadly address the following functionalities viz.,
- Capturing of a user profile for the purpose of business
- Validating checks and conditions before initiating sales or order
- Checking the inventory of items
- Fulfillment of order for items
- Management of customer issues and requests
By taking into account the above functionalities, we can evolve test scenarios or test conditions as stated below:
Comparison with known standards-Template
| Scenario-ID | Scenario-Description | Requirement-ID | Business-Usecase-ID |
|---|---|---|---|
| Scenario##### | Check whether CRM application captures details of customer | Req#### | Usecase# |
| Scenario##### | Check if CRM application validates credit worthiness of customer before initiating sales | Req#### | Usecase# |
| Scenario##### | Check if CRM application validates credit worthiness of customer before initiating sales | Req#### | Usecase# |
| Scenario##### | Check if equipment ordered is in inventory of items | Req#### | Usecase# |
| Scenario##### | Check if the geographical area in which customer lives is covered by mobile network | Req#### | Usecase# |
| Scenario##### | Check if a trouble ticket is raised for every customer issue | Req#### | Usecase# |
| Scenario##### | Check if the customer issue is handled and closed by CRM app | Req#### | Usecase# |
Comparison of specific features-Template
| Scenario-ID | Scenario-Description | Requirement-ID | Business-Usecase-ID |
|---|---|---|---|
| Scenario##### | Check the price of application wrt other software products | Req#### | Usecase# |
| Scenario##### | Check the time taken to process user requests. Compare with other software products | Req#### | Usecase# |
| Scenario##### | Check the maximum network load the application can support. Compare with other software products | Req#### | Usecase# |
| Scenario##### | Check the look and feel of a User interface. Compare with other software products | Req#### | Usecase# |
| Scenario##### | Check the end to end integration of application as compared to other software products | Req#### | Usecase# |
Note that the templates illustrate test conditions and not the detailed step-by-step description as seen in a test case.
How comparison testing can help the business
An unambiguous comparison test criteria and accurate test results can help the business, make claims for the software product like
- The quickest app with respect to response time
- The most durable product with respect to network load and so on
Test results can be used not only for promoting the software product but also to expose pitfalls and improvise the product.
An insight into the challenges, limitations, and scope of this testing:
The success of any new venture or software product is a result of various activities like design, development, testing, sales and marketing strategies, investments, and accrued profits.
In this context, Comparison testing aids in making critical decisions about the software product but cannot ensure the success of the product. In spite of exhaustive testing, the business may still fail because of inaccurate business strategies and decisions. Therefore, market research and evaluation of various business strategies is a subject by itself and beyond the scope of comparison testing.
A typical case study to understand the scope of this testing:
The launch of Disney mobile in the U.S. way back in 2005 is a case worth studying. Disney made its foray into the business of wireless services with no prior experience in Telecom. The new mobile venture stumbled very badly in the U.S. despite the brand name called “Disney”.
A postmortem into its initial failure revealed that the product failed, not because of bad design or inaccurate testing but because of bad marketing and business decisions.
Disney mobile targeted kids and sports lovers as the customers with the promise of providing unique downloading and family control features.
The same Disney mobile app which failed miserably in the U.S. gained momentum in Japan. Interestingly, this time, the main target customers were not kids but women in their ’20s and 30s.
Conclusion
Introducing a new software product is like treading into unfamiliar territory with diverse possibilities.
Many products are successful because their creators identified an unmet need in the market and understood the viability of the new idea.
Comparison testing can become a powerful tool to understand the viability of a software product.
It provides crucial business inputs to promote the software product and also exposes loopholes before the product is rolled into the market.
Please share your thoughts/suggestions in below comment section.