Multivariate Testing and A/B Testing Simplified: How to Test Webpage Optimization Techniques

There are many websites and applications that we often use. While we use them, we probably do not pay a lot of attention on how the website came to be its current version- on the other hand, if something does not seem right, we probably will never visit the site/app back again.

The question is – What does it take to get “it” right?

The “it” most often is the functionality – which we have rock solid QA processes to test and evaluate. But the “it” also consists of design, combination of elements, placement of contents on a page, sometimes even colour, orientation etc. that play a prominent role in the overall acceptance of the product by its end user.

A branch of testing that can help a great deal in this area is the Multivariate Testing and A/B testing.

multivariate testing and ab testing

In today’s article we will discuss Multivariate (MVT) testing and A/B testing types in detail.

Both these are targeted at web page optimization and improving conversion rate (the rate at which visitors becomes customers or returning visitors- in turn business) for a website.

What is Multivariate Testing?

Let us begin with an example.

If a certain website is working on designing/redesigning/determining the effectiveness of a page that should have an image and the corresponding text- After careful consideration and deliberation if the company short lists the following two images and two sentences- the possible combinations of them could be as follows:

1) Image 1

MVT testing 1

2) Image 2

mvt testing 2

3) Headline/Sentence 1: “The Goal Must be ZERO Accidents”

4) Headline/Sentence 2: “Our AIM: No ACCIDENT”

Combinations:

mvt testing 3

In the above example we have tested variations to the combinations of the fields to see which one is a good fit. Simply put, that right there is Multivariate testing.

More technically and specifically, the below formula is used to determine the no. of possible combinations that are required to test the different combinations and that is:

[# Variations on Element A] X [# Variations on Element B]….. = [Total # Variations]

In the above example, there are 2 variations for the Headline as well as 2 variations for the Image.

Thus as per the formula, there is a total of 4 combinations of the variations to be tested concurrently to find the best variation combination.

  • The main aim of performing Multivariate testing is to measure and determine the effectiveness of each variation combination on the final system.
  • After finalizing the variation combinations, testing to determine the most successful design is initiated once enough traffic is received by site.
  • The results obtained with each variation combination are compared to the others to find out which design is best suited to reach the ultimate goal (in most cases it is Sales).
  • These statistics give a clear picture of whether the particular change has been helpful or not.
  • Also, positive or negative impact on user’s interaction can also be analyzed

This entire process of continuous multivariate testing, improving design based on the results obtained, and achieving the business goals due to that (E.g.: longer engagement time for a user on a certain page) is called Landing Page Optimization whose goal is to bring more users and keep them engaged on a certain page.

This process largely involves testing with multiple variations, gathering statistics and making changes based on the values/results obtained.

Types of MVT testing:

Based on the distribution of traffic to multiple variation versions, there are multiple types of Multivariate testing that can be performed:

a) Full Factorial Testing:

It is the most preferred form of MVT testing in which every possible element variation combination is tested equally by diverting website traffic to it until a winner is found. The best thing about this method is that, there are no assumptions and it is based on hard numbers/statistics which makes it very reliable and most recommended.

The only demerit is the traffic. With the increase in the number of variation combinations, a lot of website traffic is required to analyze the data and decide the winner.

b) Fractional Factorial Testing:

As the name indicates only a fraction of all variation combination versions is exposed to website traffic. Static mathematical calculations and analysis is done for the rest of the combinations to find the best conversion rate.

Taguchi Method is the most popular method for fractional factorial multivariate testing. This method gives a less accurate result as only a sample of the variations is tested and not all. Although this method takes less time to analyze the winner, the result can never be considered as accurate as it can be in case of Full Factorial testing.

c) Adaptive Multivariate Testing:

This is a new approach in Multivariate testing. In this case, the real time response of the visitors on the webpage is analyzed to determine the best variation combination version.

Let us move further to an important question: Can Web marketing be optimized by Multivariate testing?

The answer is a resounding “Yes”.

mvt testing 4

Using Multi variate testing we can clearly determine what should be implemented and what is to be avoided. Everything is focused on the visitor’s experience.

The following aspects are considered when Multivariate testing is to be carried out:

#1. The prerequisite for Multivariate testing is to: Define marketing objectives or examine goals for the website. The below are few examples:

  • Make maximum revenue/profits through advertising, selling products, pay for clicks.
  • Create brand awareness in the clientele
  • Save expenditure- e.g.: Guide users to self-service through FAQs instead of online, in person service.

#2. Only those things should be tested that truly target the marketing objectives of the organization.

#3. Choose only those elements that will accurately measure the marketing objectives.

Examples could be:

------------

  • To make more money, the pages that include options like ‘Buy Now’/ ‘payment’/ flows to complete registration or sign up should be focused on.
  • To create awareness for visitors and promotion, ‘send to a friend’/ ‘refer a friend’/ ‘share’, etc can come in handy
  • For savings focus could be on elements like FAQs, help, contact, call, ‘Add to Cart’ clicks that leads to thank you pages, etc.

mvt testing 5

A word of caution – Beware of the following things when performing Multivariate testing:

Mistakes that should be avoided:

  1. Improper choice of variants. For e.g. suppose we change the font size, colour and style of the headline text at the same time under one version of variation combination. It will then be difficult to analyze from the data received about which variation of the headline (whether font size, colour or style) made the visitor respond differently.
  2. Too short a span of Multi variate test run. Ending the test run early and picking a small range of data to analyze the winner may lead to invalid statistics.
  3. Too long a span of Multi variant test run. Running the test too long to analyze the marginal data also leads to much wastage of time
  4. Wrong understanding of Key Indicators. Focusing, analyzing and tracking the variation combination of those indicators that are insignificant or unrelated to the end goal
  5. Only few Key Performance indicators are identified while many others are not tracked
  6. Deciding the type of visitor’s traffic to webpage. This can be very risky and problematic as not all visitors are the same.
  7. Not analyzing the results and making the right changes to the site.

mvt testing 6

Do’s and Don’ts

From the above list, a summary of do’s and Don’ts could be:

Don’ts:

Don’t try to include lot of variables in the test. The more the number of variables to test; the greater will be the combinations, which in turn means more traffic is required to gather significant statistics.

Do’s:

  1. Preview all the variation combination versions before starting test run because some of them could be incompatible or illogical. For example, one of the variable combination is of Headline saying “50% off on subscription” and off button saying “Free subscription”. Such ones should be removed.
  2. Decide the impact of combination versions on conversion rate. Including only those combinations that have a higher impact on conversion rate is a good idea.
  3. Estimate the webpage traffic in order to collect significant statistical data. Before starting test run, it is better to have a clear idea of the webpage traffic. If a webpage has only 100-200 visitors per day, then we should consider few variables only for executing multivariate test.

Pros and Cons

Till now, we have seen what is multivariate testing, how it is done, errors, factors, do’s and don’ts etc. Now, let us look at some Pros and Cons of it:

mvt testing 7

Pros:

  1. Better insight and understanding of the effect of variables or elements on conversion rate More Traffic leads to more statistical data which in turn leads to better analysis and decision making in terms of the best variable combination to reach the ultimate goal.
  2. In terms of design and layout changes, multivariate testing is flexible.

Cons:

  1. Multivariate test runs take longer time to complete.
  2. Lots of webpage traffic is required to get significant statistics.
  3. More complicated to setup test runs.
  4. Requires more number of variable combination versions for test run.

A/B Testing

That being a brief gist of all things Multivariant testing, there is no end to the variety of tests that can be done to perform webpage optimization and another popular method available is the A/B testing.

What is A/B testing?

AB testing 1

[image source]

A/B testing is also known as Split Testing.

In A/B testing, two versions of the same webpage are put under test with equal amount of webpage traffic. The version which gets maximum number of conversion is the ultimate winner. This new version definitely increases the conversion rate.

A/B Split Testing Example:

Let us understand working of A/B testing with a small example:

AB testing 2

The above image is of a webpage for Safety awareness.

This image consists of a grey button saying “Take a Quiz and win exciting prizes” .This original webpage is considered as ‘A Version’. Now ‘B version’ is designed with a variation in colour of the button from Grey to Red.

This is shown in image below:

AB testing 3

Live webpage traffic is diverted to both versions. After enough visitors have taken the test and with the statistical data received, it can be easily determined which version has higher effect on conversion rate.

Here in the above example, button saying “Take a Quiz and win exciting prizes” in Red colour attracted more visitors to hit the button and take a quiz than the older Grey button.

Thus the webpage’s ultimate goal to increase more revenue was achieved.

Pros and Cons of A/B tests:

Pros:

  1. Easy and simple method to set up experiments for webpage optimization.
  2. Reliable and accurate results can be easily determined even with small webpage traffic.
  3. Tests can be performed very quickly and statistical data can be analyzed to reach ultimate goal.
  4. Not much dependent on any form of technology.
  5. More apt for changes in layout, content, design of any webpage.

Cons:

  1. Only few or say limited number of changes can be done to a webpage at a time.
  2. It is not possible to determine the impact of different variables present on webpage on each other.

Comparison Table:

A/B testing vs Multivariate Testing

A/B TestingMultivariate Testing
Webpage traffic is split among two or more completely different versions of a webpage.Few key variables are determined and their combination is done to create versions.
Relatively less traffic is required in A/B split testing.Multi variant testing method requires huge traffic.
A/B testing method is best suited for redesigning the webpage with different ideas leading to increased conversion rate.Multivariate testing is optimizing an existing webpage without doing much or redesigning.

Conclusion:

Both methods, A/B and Multivariate testing increase conversion rate, improve performance and optimize WebPages. Both are useful in their own way and come with their unique short comings and challenges too- it is up to us to identify and analyse which method will best suit the requirement.

We, testers are mainly involved in testing the changes done to implement Multivariate or A/B tests. Once these changes are made and tested, those can be run on the production environment by the marketing or business team to gather the results.

So, it is very important that testers test these changes very carefully, otherwise the final results would be inaccurate, resulting in huge business losses as this is most of the time related to business revenue directly.

Recommended reading

6 comments ↓

#1 Sumit Mahajan

good one. is this testing manual only? is there any automation tool for this?

#2 Srijeet

thank you for sharing. learned new thing again.

#3 Ramuk

How to perform a A/B test?

#4 Nehal

Good Article and thanks for sharing.

#5 NAGIREDDY

Nice article. Highlighted the rarely known testing type.

#6 swapna

thanks for sharing .Before i dont know these testing types .

Leave a Comment