What is Comparison Testing (Learn with Examples)

Comparison testing, an oft-repeated phrase and a type of testing that evokes our attention. Let's get into the details of how the comparison test is performed and what it actually means in real time.

What is Comparison Testing?

Comparison testing is all about assessing the strengths and weaknesses of a software product with respect to other software products existing in the market. The goal of comparison testing is to provide pivotal and critical information to the business to unravel the software product's competitive advantage in the market Vis-a-vis loopholes.

Comparison Testing

What kind of comparison we do depends on the object of testing. For example, the object of testing could be anything like:

  • A web application
  • ERP application
  • CRM application
  • A module of an application requiring validation of data after completion of a transaction and so on

Establishing Criteria for Comparison Testing

Establishing criteria for comparison test for a particular software product is a subjective matter determined by the type of software application being tested and use-cases specific to the business. The test scenarios we develop are dependent on the type of application and business specific use-cases.

Testing efforts and procedures are always organized in such a way that wherever there is ambiguity, a definite strategy is evolved that can be applied across all projects.

We would, therefore, distribute this testing into two distinct phases

Phases

This testing can be performed in two distinct phases:

  • Comparing software product against known standards or benchmarks
  • Comparing software product with specific features of other existing software products

a) For Example, if a Siebel CRM application is being tested, we know that any CRM application has modules that broadly deal with capturing customer details, processing customer orders, managing customer requests and customer issues.

In the first phase of testing, we can test the functionality of the application against known standards and functionality as existing in the market at the time of testing.

We can ask questions like:

  • Does the application have all modules which a CRM application should have?
  • Do the modules perform basic functionality as expected?

We will evolve test scenarios in such a way that the test results validate the functionality of the application as against already known standards in the market.

b) In the second phase of testing, we can compare the features of an application as against the features of other software products in the market.

For Example, the following features can be considered for comparison with other software products.

#1) Price

#2) Performance of application

Example: Response time, network load

#3) User Interface (look and feel, ease of use)

In both phases of testing, testing efforts are structured in such a way that the potential areas that can cause disruption to business are identified. An appropriate testing strategy is evolved to direct test design and test execution.

Thorough knowledge of business use-cases and requirements are inevitable.

Comparison Testing Flow

The Structured Way of Performing Comparison Test

Examples of Test Scenarios for a CRM Application

Let us take the example of a CRM application for the purchase of mobile for the purpose of test scenarios.

We know that any such CRM application should broadly address the following functionalities viz.,

  • Capturing of a user profile for the purpose of business
  • Validating checks and conditions before initiating sales or order
  • Checking the inventory of items
  • Fulfillment of order for items
  • Management of customer issues and requests

By taking into account the above functionalities, we can evolve test scenarios or test conditions as stated below:

Comparison with known standards-Template

Scenario-ID

Scenario-Description

Requirement-IDBusiness-Usecase-ID
Scenario#####

Check whether CRM application captures details of customer
Req####

Usecase#

Scenario#####

Check if CRM application validates credit worthiness of customer before initiating sales
Req####

Usecase#

Scenario#####

Check if CRM application validates credit worthiness of customer before initiating sales


Req####

Usecase#

Scenario#####

Check if equipment ordered is in inventory of items
Req####

Usecase#

Scenario#####

Check if the geographical area in which customer lives is covered by mobile network
Req####

Usecase#

Scenario#####

Check if a trouble ticket is raised for every customer issueReq####

Usecase#

Scenario#####

Check if the customer issue is handled and closed by CRM appReq####

Usecase#

Comparison of specific features-Template

Scenario-ID

Scenario-Description

Requirement-IDBusiness-Usecase-ID
Scenario#####

Check the price of application wrt other software products

Req####

Usecase#

Scenario#####

Check the time taken to process user requests. Compare with other software productsReq####

Usecase#

Scenario#####

Check the maximum network load the application can support. Compare with other software productsReq####

Usecase#

Scenario#####

Check the look and feel of a User interface. Compare with other software productsReq####

Usecase#

Scenario#####

Check the end to end integration of application as compared to other software products

Req####

Usecase#

Note that the templates illustrate test conditions and not the detailed step by step description as seen in a test case.

How comparison testing can help the business

An unambiguous comparison test criteria and accurate test results can help the business, make claims for the software product like

  • The quickest app with respect to response time
  • The most durable product with respect to network load and so on

Test results can be used not only for promoting the software product but also to expose pitfalls and improvise the product.

An insight into challenges, limitations, and scope of this testing:

The success of any new venture or a software product is a result of various activities like design, development, testing, sales and marketing strategies, investments and accrued profits.

In this context, Comparison testing aids in making critical decisions about the software product but cannot ensure the success of the product. In spite of exhaustive testing, the business may still fail because of inaccurate business strategies and decisions. Therefore, market research and evaluation of various business strategies is a subject by itself and beyond the scope of comparison testing.

A typical case study to understand the scope of this testing:

The launch of Disney mobile in the U.S. way back in 2005 is a case worth studying. Disney made its foray into the business of wireless services with no prior experience in Telecom. The new mobile venture stumbled very badly in the U.S. despite the brand name called “Disney”.

A postmortem into its initial failure revealed that the product failed, not because of bad design or inaccurate testing but because of bad marketing and business decisions.

Disney mobile targeted kids and sports lovers as the customers with the promise of providing unique downloading and family control features.

The same Disney mobile app which failed miserably in the U.S. gained momentum in Japan. Interestingly, this time, the main target customers were not kids but women in 20's and 30's.

Conclusion

Introducing a new software product is like treading into unfamiliar territory with diverse possibilities.

Many products are successful because their creators identified an unmet need in the market and understood the viability of the new idea.

Comparison testing can become a powerful tool to understand the viability of a software product.

It provides crucial business inputs to promote the software product and also expose loopholes before the product is rolled in the market.

Please share your thoughts/suggestions in below comment section.