A/B Testing and its Pillars

A/B testing is a data-driven method to find out who is the winner among A and B. It’s a testing strategy where we can test 2 different hypothesis and get to know which is the best performing hypothesis that can enhance the conversion rates. This testing method is one of the weapons that is required for a company to make better decisions and uplift their growth. But, there can be multiple pitfalls that tag along if we don’t have enough data, don’t use proper statistics, do not know when to declare the winner of the test etc.

The biggest promise of A/B testing experimentation is to put the effectiveness on top. Let it be positive or negative but getting an impact on what we’ve proposed in important. The CXL institute course helps us to understand the most important pillars that are included in an A/B testing method.

The pillars of A/B testing are:

1. Introduction
2. Planning Pillar
3. Execution Pillar
4. Results Pillar
5. Outro
6. Bonus

Each of these pillars is further classified into multiple sections. Let’s dive into the gist of all the most important factors that revolve around an A/B test.

Intro

The Intro is nothing but what we need to keep in mind before jumping into the A/B testing mastery. It’s really important to understand when and why Marketers started adapting A/B testing methods. It’s like reverse psychology. Some of us know what an A/B test is and why we’re using it but knowing when and why was this test adopted would trigger the in-depth knowledge about at what point should we choose to do the test. So, the introduction section mainly includes 3 parts and they are:

1. History of A/B testing
2. The Value of A/B testing
3. When do we have to use it?

Starting with the history, it was in 2010 that VWO and Optimizely launched the A/B testing software and all of a sudden every digital enthusiast could quite easily launch their digital experiments. And thereafter, the industry emerged with new and progressive techniques. When it comes to the value of A/B testing, the real value of the test is that it helps you to make better trustworthy decisions. And before we decide when to use the test, it’s really important to know whether we have enough data to conduct the A/B test. “Data rules above all”. One of the other things to keep in mind while using the test is to make sure that your deployment does not have a negative impact on the Key Performance Indicator show measuring.

Planning pillar

The planning pillar is all about Hypothesis, KPI’s, Research, Power and significance. The effective model that is adopted in the planning pillar is the ROAR Model which signifies Risk, Optimization, Automation and Re-think. The ROAR model provides information about how much data should we have to proceed with an A/B test.

For example, Just forget about running an A/B test if you’re below 1000 conversions per month. It would be really difficult to find the winner from the pond.

First of all, the most important part of the planning pillar is the Hypothesis Setting. If there isn’t a hypothesis, there isn’t a valuable test. A hypothesis gets everyone aligned to describe the problem, have a proposed solution and predict the outcome. This can also save time on having discussions during and after the experiment. The hypothesis makes sure that you’re going in the right direction doing research, experimentation and coming up with general theories.

Next up, When to pick what kind of KPI is something to keep an eye on. Do we know whether it would be the clicks, transactions or lifetime value that can determine the winner? What about the overall criterion? KPI is a goal metric and is not generally used for A/B testing. But, we’re just trying to change the behaviour to get more clicks, transactions etc.

So, let’s see how the KPI cone looks like.

The least important is the “Clicks”. It’s not a big deal to change the clicking behaviour. The “Potential Lifetime Value” is the Golden metric which gives you some indications about a lifetime value.

Once we finish analysing the KPI’s, it’s important to do some research for more quality and the higher winning percentage of the A/B tests. This includes the ultimate “6V Research Model”. This model is used to generate user behaviour insights. The 6 V’s are:

1. View of the Customer — The Analytics behaviour
2. Voice of the Customer — Talking to the customer service
3. Versus — Finding out the competitors
4. Validated — Getting insights of previous tests
5. Verified — Searching through scientific pieces of literature
6. Value — Knowing the company

I’ll dive into a detailed version of the 6V Model in a different chapter. So, what is the role of Power and Significance here?

The Statistical power is nothing but the likelihood that an experiment will detect an effect when there is an effect there to be detected. If we’ve created something that really makes an effect, isn’t it logical to make sure that the effect needs to be detected? Thus the planning pillar gives us a success formula which is Relevant Location of the test X Relevant hypothesis X Chance on effect.

Execution Pillar

Once we have the planning done, there are certain things to keep in mind to execute the experiment and they are :

1. Designing of an A/B test
2. Developing an A/B test
3. Quality Assuring an A/B test
4. Configuring A/B test in your tool
5. Calculate the length of your A/B test
6. Monitoring the A/B test

The change you’re making should be visible, scannable and usable. Do not use “What you see is what you get “ code editor because it’s an experiment. If it works, it works. So, why measure in your own analytics tool? It’s because its the best implementation analytics solution and you have the ultimate control over your experiment.

Results Pillar

The results pillar helps you to understand when your A/B test is a winner and what to do when your A/B test is not a winner. Before starting to analyse the outcome, we need to know the below-mentioned details.

1. The test duration
2. How to isolate the test population — Does the test have an impact on all the population or a set of few users
3. Test Goals and how to isolate those users

Once we start the analysis,

1. Analyse in the analytics tool and not in the test tool
2. Avoid Sampling
3. Analyse users and not sessions
4. Analyse users who have converted and the total conversions
5. Check if the population of users that have seen the test are about the same per variation.

When we have the analysed report, we need to focus on what information is valuable to present to which group of people. We should get the ability to create an A/B test outcome template that leads to action which means there should have been an impact.

Outro

As we near the end of this testing method, one of the key aspects to keep in mind is to maintain a balance between quantity and quality while scaling up A/B testing. Both Quality and Quantity are the centres of Excellence. We need to prioritize what to work on and understand when to increase the number of tests and what optimization do we need to apply on each test. Optimization is all about effectiveness and an organization focuses on efficiency. It’s our responsibility to combine effectiveness and efficiency to put the best out there.

Bonus

This section is just about knowing what if we have more data to use. If there’s a 20% increase in the number of users, would it affect the test?

Based on some analytics, it’s calculated that if there’s an increase in the number of users, obviously the conversion might uplift and there would be more noise in the returning users. ( Returning users are people who visit your website for the first time and leave after performing a particular action and then return to the same website later).