There are many websites and applications that we often use. While we use them, we probably do not pay a lot of attention on how the website came to be its current version- on the other hand, if something does not seem right, we probably will never visit the site/app back again.
The question is – What does it take to get “it” right?
What You Will Learn:
- Multivariate Testing and A/B testing
- What is Multivariate Testing?
- Types of MVT testing:
- How to do Multivariate Testing
- Mistakes that should be avoided
- Do’s and Don’ts
- Pros and Cons
- A/B Testing
- Pros and Cons of A/B tests:
- A/B testing vs Multivariate Testing vs Split testing
- Multivariate Testing
- A/B / Split / Multivariate Testing tools
- Recommended Reading
Multivariate Testing and A/B testing
The “it” most often is the functionality – which we have rock solid QA processes to test and evaluate. But the “it” also consists of design, a combination of elements, placement of contents on a page, sometimes even color, orientation etc. that play a prominent role in the overall acceptance of the product by its end user.
A branch of testing that can help a great deal in this area is the Multivariate Testing and A/B testing.
Both these are targeted at web page optimization and improving conversion rate (the rate at which visitors becomes customers or returning visitors- in turn, business) for a website.
What is Multivariate Testing?
Let us begin with an Example.
If a certain website is working on designing/redesigning/determining the effectiveness of a page that should have an image and the corresponding text- After careful consideration and deliberation if the company shortlists the following two images and two sentences- the possible combinations of them could be as follows:
1) Image 1
2) Image 2
3) Headline/Sentence 1: “The Goal Must be ZERO Accidents”
4) Headline/Sentence 2: “Our AIM: No ACCIDENT”
In the above example, we have tested variations to the combinations of the fields to see which one is a good fit. Simply put, that right there is Multivariate testing.
More technically and specifically, the below formula is used to determine the no. of possible combinations that are required to test the different combinations and that is:
[# Variations on Element A] X [# Variations on Element B]….. = [Total # Variations]
In the above example, there are 2 variations for the Headline as well as 2 variations for the Image.
Thus as per the formula, there is a total of 4 combinations of the variations to be tested concurrently to find the best variation combination.
- The main aim of performing Multivariate testing is to measure and determine the effectiveness of each variation combination on the final system.
- After finalizing the variation combinations, testing to determine the most successful design is initiated once enough traffic is received by the site.
- The results obtained with each variation combination are compared to the others to find out which design is best suited to reach the ultimate goal (in most cases it is Sales).
- These statistics give a clear picture of whether the particular change has been helpful or not.
- Also, a positive or negative impact on user’s interaction can also be analyzed
This entire process of continuous multivariate testing, improving design based on the results obtained, and achieving the business goals due to that (Example: longer engagement time for a user on a certain page) is called Landing Page Optimization – whose goal is to bring more users and keep them engaged on a certain page.
This process largely involves testing with multiple variations, gathering statistics and making changes based on the values/results obtained.
Not only limited to websites, but multivariate testing is also required for mobile apps. Websites and mobile apps are composed of combinations of variable elements and hence, the multivariate testing is done to figure out that which combination of variations work out best.
This plays a crucial role in internet marketing strategy.
Types of MVT testing:
Based on the distribution of traffic to multiple variation versions, there are multiple types of Multivariate testing that can be performed:
a) Full Factorial Testing:
It is the most preferred form of MVT testing in which every possible element variation combination is tested equally by diverting website traffic to it until a winner is found. All possible combinations are given an equal probability. The best thing about this method is that there are no assumptions and it is based on hard numbers/statistics which makes it very reliable and most recommended.
The only demerit is the traffic. With the increase in the number of various combinations, a lot of website traffic is required to analyze the data and decide the winner.
b) Fractional or Partial Factorial Testing:
As the name indicates only a fraction of all variation combination versions is exposed to website traffic. Static mathematical calculations and analysis are done for the rest of the combinations to find the best conversion rate.
Taguchi Method is the most popular method for fractional factorial multivariate testing. This method gives a less accurate result as only a sample of the variations is tested and not all. Although this method takes less time to analyze the winner, the result can never be considered as accurate as it can be in case of Full Factorial testing.
c) Adaptive Multivariate Testing:
This is a new approach to Multivariate testing. In this case, the real-time response of the visitors on the webpage is analyzed to determine the best variation combination version.
d) Discrete Choice:
This method uncovers the interaction effects, say how people create tradeoff from the perspective of a purchase decision. It is a complex technique which systematically varies the attributes or content elements.
e) Optimal Design:
This method includes iterations and wave of testing. In addition to a testing the maximum number of creative variations in minimum time, it also permits marketers to consider relationships, interactions, and limitations across content elements of a website or app. This helps in finding the optimal solution.
Let us move further to an important question: Can Web marketing be optimized by Multivariate testing?
The answer is a resounding “Yes”.
Using Multivariate testing we can clearly determine what should be implemented and what is to be avoided. Everything is focused on the visitor’s experience.
The following aspects are considered when Multivariate testing is to be carried out:
#1. The prerequisite for Multivariate testing is to: Define marketing objectives or examine goals for the website. The below are a few examples:
- Make maximum revenue/profits through advertising, selling products, pay for clicks.
- Create brand awareness in the clientele
- Save expenditure- e.g.: Guide users to self-service through FAQs instead of online, in personal service.
#2. Only those things should be tested that truly target the marketing objectives of the organization.
#3. Choose only those elements that will accurately measure the marketing objectives.
Examples could be:
- To make more money, the pages that include options like ‘Buy Now’/ ‘payment’/ flows to complete a registration or sign up should be focused on.
- To create awareness for visitors and promotion, ‘send to a friend’/ ‘refer a friend’/ ‘share’, etc can come in handy
- For savings focus could be on elements like FAQs, help, contact, call, ‘Add to Cart’ clicks that lead to thank you pages, etc.
How to do Multivariate Testing
1. Identify a Problem
The first step is identifying the issue. This gives you the scope of improvement for your website or app. For example, the problem can be anything like why the website visitors are not clicking on the download button.
2. Formulate Hypothesis
Make a hypothesis for improving the webpage. For, example, the hypothesis can be that customers are not clicking the download button because its visibility is not appealing. So, by making it appealing, there would be increased downloads.
3. Create Variations
Select the factors and create variations. Suppose the two factors are ‘Download’ heading and ‘PDF producer’ link. As an example, we have below 12 variations:
4. Determine your sample size
Find out how many visitors are required on each page, how long you need to run the test, how many variations you are having and the statistical significance.
5. Test your tools
Test everything (mainly, is your web page/ app working fine) before you start running the test so that nothing venoms your test results.
6. Start driving traffic
Start driving the traffic to your variations.
7. Analyze your results
After executing the test for the significant amount of time, you get the results to analyze. An example is shown below:
The ones which are having 95% or more confidence level are statistically significant results.
8. Learn from your results
This is the last and an important step. From the multivariate test, you learn about your web pages/app and its visitors. You can use this learning for future tests.
A word of caution – Beware of the following things when performing Multivariate testing:
Mistakes that should be avoided
- Improper choice of variants. For e.g. suppose we change the font size, color and style of the headline text at the same time under one version of variation combination. It will then be difficult to analyze from the data received about which variation of the headline (whether font size, color or style) made the visitor respond differently.
- Too short a span of a Multivariate test run. Ending the test run early and picking a small range of data to analyze the winner may lead to invalid statistics.
- Too long a span of a Multivariate test run. Running the test too long to analyze the marginal data also leads to much wastage of time
- Wrong understanding of Key Indicators. Focusing, analyzing and tracking the variable combination of those indicators that are insignificant or unrelated to the end goal
- Only a few Key Performance indicators are identified while many others are not tracked
- Deciding the type of visitor’s traffic to a webpage. This can be very risky and problematic as not all visitors are the same.
- Not analyzing the results and making the right changes to the site.
Do’s and Don’ts
From the above list, a summary of do’s and Don’ts could be:
Don’t try to include a lot of variables in the test. The more the number of variables to test; the greater will be the combinations, which in turn means more traffic is required to gather significant statistics.
1. Preview all the variation combination versions before starting test run because some of them could be incompatible or illogical. For example, one of the variable combinations is of Headline saying “50% off on subscription” and off button saying “Free subscription”. Such ones should be removed.
2. Decide the impact of combination versions on conversion rate. Including only those combinations that have a higher impact on conversion rate is a good idea.
3. Estimate the webpage traffic in order to collect significant statistical data. Before starting the test run, it is better to have a clear idea of the webpage traffic. If a webpage has only 100-200 visitors per day, then we should consider a few variables only for executing the multivariate test.
Pros and Cons
Till now, we have seen what is multivariate testing, how it is done, errors, factors, do’s and don’ts etc. Now, let us look at some Pros and Cons of it:
- Better insight and understanding of the effect of variables or elements on conversion rate More Traffic leads to more statistical data which in turn leads to better analysis and decision making in terms of the best variable combination to reach the ultimate goal.
- In terms of design and layout changes, multivariate testing is flexible.
- Multivariate test runs take a longer time to complete.
- Lots of webpage traffic is required to get significant statistics.
- More complicated to set up test runs.
- Requires more number of variable combination versions for test run.
That being a brief list of all things Multivariant testing, there is no end to the variety of tests that can be done to perform webpage optimization and another popular method available is the A/B testing.
What is A/B testing?
A/B testing is also sometimes known as Split Testing. However, the split testing is different. We will see the difference between them in the later part of this tutorial.
In A/B testing, two versions of the same webpage are put under test with an equal amount of webpage traffic. The version which gets a maximum number of conversion is the ultimate winner. This new version definitely increases the conversion rate.
A/B Split Testing Example:
Let us understand the working of A/B testing with a small example:
The above image is of a webpage for Safety awareness.
This image consists of a grey button saying “Take a Quiz and win exciting prizes”.This original webpage is considered as ‘A Version’. Now ‘B version’ is designed with a variation in color of the button from Grey to Red.
This is shown in the image below:
Live webpage traffic is diverted to both versions. After enough visitors have taken the test and with the statistical data received, it can be easily determined which version has a higher effect on conversion rate.
Here in the above example, a button saying “Take a Quiz and win exciting prizes” in Red color attracted more visitors to hit the button and take a quiz than the older Grey button.
Thus the webpage’s ultimate goal to increase more revenue was achieved.
Pros and Cons of A/B tests:
- Easy and simple method to set up experiments for webpage optimization.
- Reliable and accurate results can be easily determined even with small webpage traffic.
- Tests can be performed very quickly and statistical data can be analyzed to reach the ultimate goal.
- Not much dependent on any form of technology.
- More apt for changes in layout, content, design of any webpage.
- Only a few or say a limited number of changes can be done to a webpage at a time.
- It is not possible to determine the impact of different variables present on a webpage on each other.
A/B testing vs Multivariate Testing vs Split testing
A/B testing, multivariate testing and split testing are the three main types of UX(User experience) variant testing. Let’s see how they are different from each other.
|A/B Testing||Multivariate Testing|
|Webpage traffic is split among two or more completely different versions of a webpage.||Few key variables are determined and their combination is done to create versions.|
|Relatively less traffic is required in A/B split testing.||Multi variant testing method requires huge traffic.|
|Tests only one variable to see the effect of the change.||Tests multiple variables together to see the combined change effect|
|A/B testing method is best suited for redesigning the webpage with different ideas leading to increased conversion rate.||Multivariate testing is optimizing an existing webpage without doing much or redesigning.|
The below two images gives you a very good illustrative comparison between A/B Testing and Multivariate testing.
We have another variant here known as split URL testing which is much more complicated than A/B testing and involves server-side changes, where we have two different web pages which are tested against each other. This type of testing works well for landing pages in cases where the design team needs to decide on which one will work better.
A/B / Split / Multivariate Testing tools
There are a lot of tools available in the market for these three types of UX testing. I would name a few best here which you can explore. They are Google Optimize, Optimizely, VMO, Qubit, Maxymiser and AB Tasty.
Both methods, A/B and Multivariate testing increase conversion rate, improve performance and optimize WebPages and apps. Both are useful in their own way and come with their unique shortcomings and challenges too- it is up to us to identify and analyze which method will best suit the requirement.
We, testers are mainly involved in testing the changes done to implement Multivariate or A/B tests. Once these changes are made and tested, those can be run on the production environment by the marketing or business team to gather the results.
So, it is very important that testers test these changes very carefully, otherwise, the final results would be inaccurate, resulting in huge business losses as this is most of the time related to business revenue directly.