Pairwise Testing or All-Pairs Testing Tutorial with Tools and Examples

What is Pairwise Testing and How It is Effective Test Design Technique for Finding Defects:

In this article, we are going to learn about a ‘Combinatorial Testing’ technique called ‘Pairwise Testing’ also known as ‘All-Pairs Testing’.

Smart testing is the need of the hour. 90% of the time’s system testing team has to work with tight schedules. So test design techniques should be very effective for maximum test coverage and high defect yield rate.

Pairwise Testing Or All-Pairs Testing Tutorial2

Definition: What is Pairwise Testing?

Pairwise Testing is a test design technique that delivers hundred percent test coverage.

ISTQB defines All-Pairs Testing (or Pairwise Testing) as A black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.

The output of a software application depends on many factors e.g. input parameters, state variables and environment configurations. Techniques like boundary value analysis and equivalence partitioning can be useful to identify the possible values for individual factors. But it is impractical to test all possible combinations of values for all those factors. So instead a subset of combinations is generated to satisfy all factors.

All-Pairs technique is very helpful for designing tests for applications involving multiple parameters. Tests are designed such that for each pair of input parameters to a system, there are all possible discrete combinations of those parameters. The test suite covers all combinations; therefore it is not exhaustive yet very effective in finding bugs.

Let us learn how to apply All-Pairs Testing by this example.

Pairwise Testing Example

Car Ordering Application:

  • The car ordering application allows for Buying and Selling cars. It should support trading in Delhi and Mumbai.
  • The application should have registration numbers, may be valid or invalid. It should allow the trade of following cars: BMW, Audi, and Mercedes.
  • Two types of booking can be done: E-booking and In Store.
  • Orders can be placed only during trading hours.

Step #1: Let’s list down the variables involved.

1) Order category
a. Buy
b. Sell

2) Location
a. Delhi
b. Mumbai

3) Car brand
a. BMW
b. Audi
c. Mercedes

4) Registration numbers
a. Valid (5000)
b. Invalid

5) Order type
a. E-Booking
b. In-store

6) Order time
a. Working hours
b. Non-working hours

If we want to test all possible valid combinations:
= 2 X 2 X 3 X 5000 X 2 X 2
= 240000  Valid test cases combinations 🙁

There is also an infinite number of invalid combinations.

Step #2: Let’s simplify

– Use a smart representative sample.
– Use groups and boundaries, even when data is non-discrete.
– Reduce Registration Number to Two

  1. Valid registration number
  2. Invalid registration number

Now let’s calculate the number of possible combinations
= 2 X 2 X 3 X 2 X 2 X 2
= 96

Step #3: Arranging variables and values involved.

When we arrange variables and values involved, it looks something like this.

all-pairs testing example 1

Now order the variables so that the one with the most number of values is first and the least is last.

all-pairs testing example 2

Step #4: Arrange variables to create a test suite

Let’s start filling in the table column by column. Initially, the table should look something like this. The three values of Product (variable having the highest number of values) should be written two times each (two is the number of values of next highest variable i.e. Order category).

all-pairs testing example 3

The Order Category column has two values. That’s how many times we need to insert the values of the first column, Product.

all-pairs testing example 4

For each set of values in column 1, we put both values of column 2. Repeat the same for column 3.

all-pairs testing example 5

We have a Buy and Delhi, but wait – there’s no Buy and Mumbai. We have a Sell and Mumbai, but there’s no Sell and Delhi. Let’s swap around the values in the second set in the third column.

all-pairs testing example 6

This looks much better!

We will repeat the same steps for column 3 and 4.

all-pairs testing example 7

When columns 3 and 4 are compared, each value in column 3 has both the values of column 4. But when you compare the 2nd and 4th column, we have Buy and Valid & Sell and Invalid .i.e. Buy does not have ‘Invalid’ and Sell does not have ‘Valid’. Hence we need to interchange the last set of values in the 4th column.

all-pairs testing example 8

Column 6 (Order time) is problematic. We are missing Buy/Non-working hours and Sell/Working hours. We can’t fit our missing pairs by swapping around values as we already swapped all the rows if we swap now we may miss other possible pairs which are already sorted. So, we add two more test cases that contain these pairs. Hence, the blank rows!

all-pairs testing example 9

Now we will fill in the empty cells as we desire because the other variable values are purely arbitrary (or Don’t Cares ~).

all-pairs testing example 10

Hurray! All pairs in 8 cases, instead of all combinations in 96!

Hence, we saw how efficient All-pairs technique of test design is. There stands a good chance of finding bugs and it is fun and powerful.

The pairwise testing technique has some limitations as well.

  • It fails when the values selected for testing are incorrect.
  • It fails when highly probable combinations get too little attention.
  • It fails when interactions between the variables are not understood well.

Pairwise Testing Tools:

Tools are available that applies the all-pairs testing technique that facilitates us to effectively automate the Test Case Design process by generating a compact set of parameter value choices as the desired Test Cases. Some well-known tools from the industry are:

  • PICT‘Pairwise Independent Combinatorial Testing’, provided by Microsoft Corp.
  • IBM FoCuS‘Functional Coverage Unified Solution’, provided by IBM.
  • ACTS ‘Advanced Combinatorial Testing System’, provided by NIST, an agency of the US Government.
  • Hexawise
  • Jenny
  • Pairwise by Inductive AS
  • VPTag free All-Pair Testing Tool

Conclusion:

The pairwise testing technique can dramatically reduce the number of combinations to be covered but remains very effective in terms of fault detection. It is indeed a smart test design technique that guarantees a win-win situation for both test effort and test effectiveness.

During the Test planning phase of software testing, the Pairwise testing technique should always be taken into consideration. Either we are doing it manually or using any tool to generate test cases, it becomes a necessary component of the test plan because it, in turn, affects Test estimation.

Recommended Reading

29 thoughts on “Pairwise Testing or All-Pairs Testing Tutorial with Tools and Examples”

  1. kudos for the great example. do you have more examples to share?

    Reply
  2. Clear explanation on how pairwise combinations are created. Thanks!

    Reply
  3. Gr8 explanation and nice information
    Thanks!

    Reply
  4. Nice explanation. Another example would be good!

    Reply
  5. Nice and clear explanation.. Thumbs up..:)

    Reply
  6. Really has nice concept reducing the test cases….Next Pairwise.

    Reply
  7. Good to know that thistesting is so much important that few companies have created tool for it.

    what kind of things these tools have provided apart from combination

    Reply
  8. Really useful article. I enjoyed the read 🙂

    Reply
  9. Nice article. Can you explain anyone of the pair wise testing tool widely using across the industry.

    Reply
  10. If you swap buy/sell in 2nd column, you will not to add rows in step 6, right? So why don’t you do that?

    Reply
  11. Simply and clear, It explains the concept in deep. Thank you:)

    Reply
  12. thanks for this explanation, its useful.
    what is concept of 1-wise and 2-wise pair testing? how this will be calculated ?

    Reply
  13. Why don’t we have 9th test case for Buy – invalid – non working hours

    Reply
  14. Hi,

    i am unable to download Advanced Combinatorial Testing System (ACTS) tool. Can some one help me out ? Actually i am looking for pairwise testing open source tool

    Thanks and Regrads,
    Rupali.

    Reply
  15. Thank you, it’s a great explanation, very helpful. I’m currently using PictMaster, very easy to use, I recommend it.

    Reply
  16. I understand the Test-Cases combination but I unable to understand the Logic of Pairing-Details.
    – There is no permutation combination generated in OUTPUT file by ALLPairs tool,
    – So I would request you kindly explain me Logic of “Pairing-details”

    Reply
  17. Great Example step by step

    Reply
  18. This example helped me alot.

    Reply
  19. In the real application world, this is pairwise testing is non-sense and doesn’t help at all. The application test is more for human to judge what scenario should be for the function, not a math to put all the fields together and then just remove any values without understanding what scenario and is it any important test scenario or not!! we can’t based on the maths to remove the test cases, but we need to understand what is that case and it is ok to not test it!!

    Reply
    • Ciny,

      Sorry to disagree but in my line of work it is essential. As a healthcare interoperability engineer I cannot test all variables with all possible values. But, I can craft a set of cases that will test variability within each of those combinations.

      Realize that I deal with objects that contain 5-100 individual variables with some of those variables containing thousands or tens of thousands unique values. I can limit to most frequently used and still end up with thousands of test cases if I don’t use a variant of pairwise testing.

      This doesn’t eliminate the need to do full sample sets (if sampling theory dictates) but I can do those based on risk.

      This is an effective tool, even given the limitations listed in the article, and using it will improve not only your effectiveness but your find rate. Pair this with Sampling Theory and you will know when to dig deeper.

      I will agree that you must analyze your variables — there are certainly times when you will want to test some element explicitly — but this tool helps reduce the cost of too many values and variables in a test effort.

      Reply
  20. Perfect! Now I understand it!

    Reply
  21. Brave…I already tried myself but with this reading became more clear thank you so much

    Reply
  22. Hey, thanks about this explanation. Now my understanding is far better.

    Reply
  23. Revolutional update of captchas breaking package “XEvil 5.0”:

    Captchas solution of Google (ReCaptcha-2 and ReCaptcha-3), Facebook, BitFinex, Bing, Hotmail, SolveMedia, Yandex,
    and more than 12000 another size-types of captcha,
    with highest precision (80..100%) and highest speed (100 img per second).
    You can use XEvil 5.0 with any most popular SEO/SMM programms: iMacros, XRumer, SERP Parser, GSA SER, RankerX, ZennoPoster, Scrapebox, Senuke, FaucetCollector and more than 100 of other programms.

    Interested? You can find a lot of introducing videos about XEvil in YouTube.

    Free XEvil Demo available.

    See you later 😉

    Reply
  24. Hi,
    can I ask who is author of this article and when is was written? I want to quote this source for my Bachelor thesis.

    Thank you so much,
    David

    Reply
  25. So far one of the best examples and explanation out there. Still need some time to get it in my head though. A few more examples would have helped.

    Kind regards,

    Reply
  26. Please fix your initial calculation for “all possible valid combinations”:

    The “Registration numbers” are 5001 cases, not 5000, so the calculation is
    … * 5001 * … = 240048

    Reply
  27. Good article, but i don’t understand the order type column. Tried to follow the steps thoroughly, but why are the first group substituted there? Writing them down as E-store and In-Store, i don’t understand why the first group is substituted, i don’t think it’s clearly explained.

    Reply
  28. Thank you! A good explanation of the swapping principles in rows. Thank you!

    Reply

Leave a Comment