There are many websites and applications that we often use. While we use them, we probably do not pay a lot of attention on how the website came to be its current version- on the other hand, if something does not seem right, we probably will never visit the site/app back again.
The question is – What does it take to get “it” right?
The “it” most often is the functionality – which we have rock solid QA processes to test and evaluate. But the “it” also consists of design, combination of elements, placement of contents on a page, sometimes even colour, orientation etc. that play a prominent role in the overall acceptance of the product by its end user.
A branch of testing that can help a great deal in this area is the Multivariate Testing and A/B testing.
Both these are targeted at web page optimization and improving conversion rate (the rate at which visitors becomes customers or returning visitors- in turn business) for a website.
What You Will Learn:
Let us begin with an example.
If a certain website is working on designing/redesigning/determining the effectiveness of a page that should have an image and the corresponding text- After careful consideration and deliberation if the company short lists the following two images and two sentences- the possible combinations of them could be as follows:
1) Image 1
2) Image 2
3) Headline/Sentence 1: “The Goal Must be ZERO Accidents”
4) Headline/Sentence 2: “Our AIM: No ACCIDENT”
In the above example we have tested variations to the combinations of the fields to see which one is a good fit. Simply put, that right there is Multivariate testing.
More technically and specifically, the below formula is used to determine the no. of possible combinations that are required to test the different combinations and that is:
[# Variations on Element A] X [# Variations on Element B]….. = [Total # Variations]
In the above example, there are 2 variations for the Headline as well as 2 variations for the Image.
Thus as per the formula, there is a total of 4 combinations of the variations to be tested concurrently to find the best variation combination.
This entire process of continuous multivariate testing, improving design based on the results obtained, and achieving the business goals due to that (E.g.: longer engagement time for a user on a certain page) is called Landing Page Optimization – whose goal is to bring more users and keep them engaged on a certain page.
This process largely involves testing with multiple variations, gathering statistics and making changes based on the values/results obtained.
Based on the distribution of traffic to multiple variation versions, there are multiple types of Multivariate testing that can be performed:
a) Full Factorial Testing:
It is the most preferred form of MVT testing in which every possible element variation combination is tested equally by diverting website traffic to it until a winner is found. The best thing about this method is that, there are no assumptions and it is based on hard numbers/statistics which makes it very reliable and most recommended.
The only demerit is the traffic. With the increase in the number of variation combinations, a lot of website traffic is required to analyze the data and decide the winner.
b) Fractional Factorial Testing:
As the name indicates only a fraction of all variation combination versions is exposed to website traffic. Static mathematical calculations and analysis is done for the rest of the combinations to find the best conversion rate.
Taguchi Method is the most popular method for fractional factorial multivariate testing. This method gives a less accurate result as only a sample of the variations is tested and not all. Although this method takes less time to analyze the winner, the result can never be considered as accurate as it can be in case of Full Factorial testing.
c) Adaptive Multivariate Testing:
This is a new approach in Multivariate testing. In this case, the real time response of the visitors on the webpage is analyzed to determine the best variation combination version.
Let us move further to an important question: Can Web marketing be optimized by Multivariate testing?
The answer is a resounding “Yes”.
Using Multi variate testing we can clearly determine what should be implemented and what is to be avoided. Everything is focused on the visitor’s experience.
The following aspects are considered when Multivariate testing is to be carried out:
#1. The prerequisite for Multivariate testing is to: Define marketing objectives or examine goals for the website. The below are few examples:
#2. Only those things should be tested that truly target the marketing objectives of the organization.
#3. Choose only those elements that will accurately measure the marketing objectives.
Examples could be:
A word of caution – Beware of the following things when performing Multivariate testing:
From the above list, a summary of do’s and Don’ts could be:
Don’t try to include lot of variables in the test. The more the number of variables to test; the greater will be the combinations, which in turn means more traffic is required to gather significant statistics.
Till now, we have seen what is multivariate testing, how it is done, errors, factors, do’s and don’ts etc. Now, let us look at some Pros and Cons of it:
That being a brief gist of all things Multivariant testing, there is no end to the variety of tests that can be done to perform webpage optimization and another popular method available is the A/B testing.
What is A/B testing?
A/B testing is also known as Split Testing.
In A/B testing, two versions of the same webpage are put under test with equal amount of webpage traffic. The version which gets maximum number of conversion is the ultimate winner. This new version definitely increases the conversion rate.
A/B Split Testing Example:
Let us understand working of A/B testing with a small example:
The above image is of a webpage for Safety awareness.
This image consists of a grey button saying “Take a Quiz and win exciting prizes” .This original webpage is considered as ‘A Version’. Now ‘B version’ is designed with a variation in colour of the button from Grey to Red.
This is shown in image below:
Live webpage traffic is diverted to both versions. After enough visitors have taken the test and with the statistical data received, it can be easily determined which version has higher effect on conversion rate.
Here in the above example, button saying “Take a Quiz and win exciting prizes” in Red colour attracted more visitors to hit the button and take a quiz than the older Grey button.
Thus the webpage’s ultimate goal to increase more revenue was achieved.
|A/B Testing||Multivariate Testing|
|Webpage traffic is split among two or more completely different versions of a webpage.||Few key variables are determined and their combination is done to create versions.|
|Relatively less traffic is required in A/B split testing.||Multi variant testing method requires huge traffic.|
|A/B testing method is best suited for redesigning the webpage with different ideas leading to increased conversion rate.||Multivariate testing is optimizing an existing webpage without doing much or redesigning.|
Both methods, A/B and Multivariate testing increase conversion rate, improve performance and optimize WebPages. Both are useful in their own way and come with their unique short comings and challenges too- it is up to us to identify and analyse which method will best suit the requirement.
We, testers are mainly involved in testing the changes done to implement Multivariate or A/B tests. Once these changes are made and tested, those can be run on the production environment by the marketing or business team to gather the results.
So, it is very important that testers test these changes very carefully, otherwise the final results would be inaccurate, resulting in huge business losses as this is most of the time related to business revenue directly.