Multivariate Testing and A/B Testing Simplified: How to Test Webpage Optimization Techniques

In today’s article, we will discuss Multivariate testing and A/B testing types in detail. Let’s get started.

There are a number of websites and applications that we use. While we use them, we probably do not pay a lot of attention to how the website came to be its current version. If something does not seem right, we probably will never visit the site/app again.

The question is – What does it take to get “it” right?

The “it” is the functionality, which we have rock solid QA processes to test and evaluate. But the “it” also consists of design, a combination of elements, placement of contents on a page, sometimes even color, orientation etc. that plays a prominent role in the overall acceptance of the product by its end user.

The branch of testing that can help a great deal in this area is Multivariate Testing and A/B testing. Both are targeted at web page optimization and improving conversion rates (the rate at which visitors become customers or returning visitors- in turn, business) for a website.

Multivariate Testing and A/B testing

What is Multivariate Testing

Let us begin with an Example.

If a certain website is working on designing/redesigning/determining the effectiveness of a page that should have an image and the corresponding text- After careful consideration and deliberation if the company shortlists the following two images and two sentences- the possible combinations of them could be as follows:

1) Image 1

2) Image 2

3) Headline/Sentence 1: “The Goal Must be ZERO Accidents”

4) Headline/Sentence 2: “Our AIM: No ACCIDENT”

Combinations:

In the above example, we have tested variations in the combinations of the fields to see which one is a good fit. Simply put, that right there is Multivariate testing.

More technically and specifically, the below formula is used to determine the no. of possible combinations that are required to test the different combinations and that is:

[# Variations on Element A] X [# Variations on Element B]….. = [Total # Variations]

In the above example, there are 2 variations for the Headline as well as 2 variations for the Image.

Thus, as per the formula, there are a total of 4 combinations of the variations to be tested concurrently to find the best variation combination.

• The main aim of performing Multivariate testing is to measure and determine the effectiveness of each variation combination on the final system.
• After finalizing the variation combinations, testing to determine the most successful design will be initiated once enough traffic is received by the site.
• The results obtained with each variation combination are compared to the others to find out which design is best suited to reach the ultimate goal (in most cases it is Sales).
• These statistics give a clear picture of whether a particular change has been helpful or not.
• Also, a positive or negative impact on user’s interaction can also be analyzed.

This entire process of continuous multivariate testing, improving design based on the results obtained, and achieving the business goals due to that (Example: longer engagement time for a user on a certain page) is called Landing Page Optimization – whose goal is to bring more users and keep them engaged on a certain page.

This process largely involves testing with multiple variations, gathering statistics and making changes based on the values/results obtained.

It is not limited to websites, but multivariate testing is also required for mobile apps. Websites and mobile apps are composed of combinations of variable elements and hence, multivariate testing is done to figure out which combination of variations work out best.

This plays a crucial role in internet marketing strategy.

Types of MVT testing

Based on the distribution of traffic to multiple variation versions, there are multiple types of Multivariate testing that can be performed:

a) Full Factorial Testing

It is the most preferred form of MVT testing in which every possible element variation combination is tested equally by diverting website traffic to it until a winner is found. All possible combinations are given an equal probability. The best thing about this method is that there are no assumptions and it is based on hard numbers/statistics which makes it very reliable and most recommended.

The only demerit is the traffic. With the increase in the number of various combinations, a lot of website traffic is required to analyze the data and decide the winner.

b) Fractional or Partial Factorial Testing

As the name indicates, only a fraction of all variation combination versions are exposed to website traffic. Static mathematical calculations and analysis are done for the rest of the combinations to find the best conversion rate.

Taguchi Method is the most popular method for fractional factorial multivariate testing. This method gives a less accurate result as only a sample of the variations is tested and not all. Although this method takes less time to analyze the winner, the result can never be considered as accurate as it can be in case of Full Factorial testing.

c) Adaptive Multivariate Testing

This is a new approach to Multivariate testing. In this case, the real-time response of the visitors on the webpage is analyzed to determine the best variation combination version.

d) Discrete Choice

This method uncovers the interaction effects, say how people create tradeoffs from the perspective of a purchase decision. It is a complex technique which systematically varies the attributes or content elements.

e) Optimal Design

This method includes iterations and waves of testing. In addition to testing the maximum number of creative variations in minimum time, it also permits marketers to consider relationships, interactions, and limitations across content elements of a website or app. This helps in finding the optimal solution.

Let’s move further to an important question: Can web marketing be optimized by Multivariate testing?

The answer is a resounding “Yes”.

Using Multivariate testing we can clearly determine what should be implemented and what is to be avoided. Everything is focused on the visitor’s experience.

The following aspects are to be considered when Multivariate testing is to be carried out:

#1. The prerequisite for Multivariate testing is to: Define marketing objectives or examine goals for the website. Given below are a few examples:

• Make maximum revenue/profits through advertising, selling products, pay for clicks.
• Create brand awareness in the clientele
• Save expenditure- e.g. Guide users to self-service through FAQs instead of online in personal service.

#2. Only those things should be tested that truly target the marketing objectives of the organization.

#3. Choose only those elements that will accurately measure the marketing objectives.

Examples could be:

• To make more money, the focus should be on the pages that include options like ‘Buy Now’/ ‘payment’/ flows to complete a registration or sign up.
• To create awareness for visitors and promotion, ‘send to a friend’/ ‘refer a friend’/ ‘share’, etc can come in handy.
• For savings, the focus could be on elements like FAQs, help, contacts, calls, and “Add to Cart” clicks that lead to thank you pages, etc.

How to do Multivariate Testing

1. Identify a Problem 

The first step is identifying the issue. This will give you the scope of improvement for your website or app. For example, the problem can be anything like why the website visitors are not clicking on the download button.

2. Formulate Hypothesis 

Create a hypothesis for improving the webpage. For example, the hypothesis can be that customers are not clicking the download button because its visibility is not appealing. So, by making it appealing, there would be increased downloads. 

3. Create Variations 

Select factors and create variations. Suppose the two factors are ‘Download’ heading and ‘PDF producer’ link. As an example, we have below 12 variations:

4. Determine your sample size 

Find out how many visitors are required on each page, how long you need to run the test, how many variations you have and the statistical significance. 

5. Test your tools

Test everything (mainly, is your web page/ app working fine) before you start running the test so that nothing venoms your test results.

6. Start driving traffic

Start driving traffic to your variations.

7. Analyze your results

After executing the test for a significant amount of time, you get the results to analyze. An example is shown below:

The ones which are having 95% or more confidence levels are statistically significant results. 

8. Learn from your results

This is the last and an important step. From the multivariate test, you learn about your web pages/app and its visitors. You can use this learning for future tests.

A word of caution – Beware of the following things when performing Multivariate testing:

Mistakes that should be avoided

1. Improper choice of variants. For e.g. Suppose we change the font size, color and style of the headline text at the same time under one version of variation combination. It will then be difficult to analyze from the data received about which variation of the headline (whether font size, color or style) made the visitor respond differently.
2. It’s too short a span of a Multivariate test run. Ending the test run early and picking a small range of data to analyze the winner may lead to invalid statistics.
3. Too long a span of a Multivariate test run. Running the test too long to analyze the marginal data also leads to much wastage of time
4. Wrong understanding of Key Indicators. Focusing, analyzing and tracking the variable combination of those indicators that are insignificant or unrelated to the end goal
5. Only a few Key Performance indicators are identified while many others are not tracked
6. Deciding the type of visitor’s traffic to a webpage. This can be very risky and problematic as not all visitors are the same.
7. Not analyzing the results and making the right changes to the site.

Do’s and Don’ts

From the above list, a summary of do’s and Don’ts could be:

Don’ts:

Don’t try to include a lot of variables in the test. The more the number of variables to test; the greater the combinations will be, which in turn means more traffic is required to gather significant statistics.

Do’s:

1. Preview all the variation combination versions before starting the test run because some of them could be incompatible or illogical. For example, one of the variable combinations is of Headline saying “50% off on subscription” and off button saying “Free subscription”. Such ones should be removed.

2. Decide on the impact of combination versions on conversion rates. Including only those combinations that have a higher impact on conversion rate is a good idea.

3. Estimate the webpage traffic in order to collect significant statistical data. Before starting the test run, it is better to have a clear idea of the webpage traffic. If a webpage has about 100-200 visitors per day, then we should consider a few variables only for executing the multivariate test.

Pros and Cons

Till now, we have seen what multivariate testing is, how it is done, errors, factors, do’s and don’ts etc. Now, let us look at some Pros and Cons of it:

Pros:

1. Better insight and understanding of the effect of variables or elements on conversion rate More Traffic leads to more statistical data which in turn leads to better analysis and decision making in terms of the best variable combination to reach the ultimate goal.
2. In terms of design and layout changes, multivariate testing is flexible.

Cons:

1. Multivariate test runs take a longer time to complete.
2. Lots of webpage traffic is required to get significant statistics.
3. It is complicated to set up test runs.
4. Requires a higher number of variable combination versions for the test run.

A/B Testing

That being a brief list of all things multivariant testing, there is no end to the variety of tests that can be done to perform webpage optimization and another popular method available is A/B testing.

What is A/B testing?

[image source]

A/B testing is also sometimes known as Split Testing. However, split testing is different. We will see the difference between them in the later part of this tutorial.

In A/B testing, two versions of the same webpage are put under test with an equal amount of webpage traffic. The version which gets a maximum number of conversions is the ultimate winner. This new version will definitely increase the conversion rate.

A/B Split Testing Example:

Let us understand the working of A/B testing with a small example:

The above image is of a webpage for Safety awareness.

This image consists of a grey button saying “Take a Quiz and win exciting prizes”. This original webpage is considered as “A Version.” Now the B version is designed with a variation in color of the button from Grey to Red.

This is shown in the image below:

Live webpage traffic has been diverted to both versions. After enough visitors have taken the test and with the statistical data received, it can be easily determined which version has a higher effect on conversion rate.

Here in the above example, a button saying “Take a Quiz and win exciting prizes” in Red color attracted more visitors to hit the button and take a quiz than the older grey button.

Thus the webpage’s ultimate goal to increase more revenue was achieved.

Pros and Cons of A/B tests

Pros:

1. An easy and simple method to set up experiments for webpage optimization.
2. Reliable and accurate results can be easily determined even with small webpage traffic.
3. Tests can be performed very quickly and statistical data can be analyzed to reach the ultimate goal.
4. Not much dependence on any form of technology.
5. More apt for changes in layout, content, and design of any webpage.

Cons:

1. Only a limited number of changes can be made to a webpage at a time.
2. It is not possible to determine the impact of different variables present on a webpage on each other.

Comparison Table

A/B testing vs Multivariate Testing vs Split testing

A/B testing, multivariate testing and split testing are the three main types of UX(User experience) variant testing. Let’s see how they are different from each other.

The below two images give you a very good illustrative comparison between A/B Testing and Multivariate testing.

Multivariate Testing

Split Testing

 We have another variant here known as split URL testing which is much more complicated than A/B testing and involves server-side changes, where we have two different web pages which are tested against each other. This type of testing works well for landing pages in cases where the design team needs to decide on which one will work better.

A/B / Split / Multivariate Testing tools

There are a lot of tools available in the market for these three types of UX testing. I will name a few of the best here which you can explore. They are Google Optimize, Optimizely, VMO, Qubit, Maxymiser and AB Tasty.

Conclusion

Both the methods, A/B and Multivariate testing increase conversion rates, improve performance and optimize WebPages and apps. Both are useful in their own way and come with their unique shortcomings and challenges too- it is up to us to identify and analyze which method will best suit the requirement.

We as testers are mainly involved in testing the changes done to implement Multivariate or A/B tests. Once these changes are made and tested, these can be run in the production environment by the marketing or business team to gather the results.

So, it is very important that testers test these changes very carefully, otherwise, the final results would be inaccurate, resulting in huge business losses as this is most of the time related to business revenue directly.

Feel free to post your thoughts, doubts, queries and other feedback in the comments section below. We would love to hear from you.