A/B Testing and its Pillars

A/B testing is a data-driven method to find out who is the winner among A and B. It’s a testing strategy where we can test 2 different hypothesis and get to know which is the best performing hypothesis that can enhance the conversion rates. This testing method is one of the weapons that is required for a company to make better decisions and uplift their growth. But, there can be multiple pitfalls that tag along if we don’t have enough data, don’t use proper statistics, do not know when to declare the winner of the test etc.

The biggest promise of A/B testing experimentation is to put the effectiveness on top. Let it be positive or negative but getting an impact on what we’ve proposed in important. The CXL institute course helps us to understand the most important pillars that are included in an A/B testing method.

The pillars of A/B testing are:

  1. Introduction
  2. Planning Pillar
  3. Execution Pillar
  4. Results Pillar
  5. Outro
  6. Bonus

Each of these pillars is further classified into multiple sections. Let’s dive into the gist of all the most important factors that revolve around an A/B test.


  1. History of A/B testing
  2. The Value of A/B testing
  3. When do we have to use it?

Starting with the history, it was in 2010 that VWO and Optimizely launched the A/B testing software and all of a sudden every digital enthusiast could quite easily launch their digital experiments. And thereafter, the industry emerged with new and progressive techniques. When it comes to the value of A/B testing, the real value of the test is that it helps you to make better trustworthy decisions. And before we decide when to use the test, it’s really important to know whether we have enough data to conduct the A/B test. “Data rules above all”. One of the other things to keep in mind while using the test is to make sure that your deployment does not have a negative impact on the Key Performance Indicator show measuring.

Planning pillar

For example, Just forget about running an A/B test if you’re below 1000 conversions per month. It would be really difficult to find the winner from the pond.

First of all, the most important part of the planning pillar is the Hypothesis Setting. If there isn’t a hypothesis, there isn’t a valuable test. A hypothesis gets everyone aligned to describe the problem, have a proposed solution and predict the outcome. This can also save time on having discussions during and after the experiment. The hypothesis makes sure that you’re going in the right direction doing research, experimentation and coming up with general theories.

Next up, When to pick what kind of KPI is something to keep an eye on. Do we know whether it would be the clicks, transactions or lifetime value that can determine the winner? What about the overall criterion? KPI is a goal metric and is not generally used for A/B testing. But, we’re just trying to change the behaviour to get more clicks, transactions etc.

So, let’s see how the KPI cone looks like.

Image for post
Image for post

The least important is the “Clicks”. It’s not a big deal to change the clicking behaviour. The “Potential Lifetime Value” is the Golden metric which gives you some indications about a lifetime value.

Once we finish analysing the KPI’s, it’s important to do some research for more quality and the higher winning percentage of the A/B tests. This includes the ultimate “6V Research Model”. This model is used to generate user behaviour insights. The 6 V’s are:

  1. View of the Customer — The Analytics behaviour
  2. Voice of the Customer — Talking to the customer service
  3. Versus — Finding out the competitors
  4. Validated — Getting insights of previous tests
  5. Verified — Searching through scientific pieces of literature
  6. Value — Knowing the company

I’ll dive into a detailed version of the 6V Model in a different chapter. So, what is the role of Power and Significance here?

The Statistical power is nothing but the likelihood that an experiment will detect an effect when there is an effect there to be detected. If we’ve created something that really makes an effect, isn’t it logical to make sure that the effect needs to be detected? Thus the planning pillar gives us a success formula which is Relevant Location of the test X Relevant hypothesis X Chance on effect.

Execution Pillar

  1. Designing of an A/B test
  2. Developing an A/B test
  3. Quality Assuring an A/B test
  4. Configuring A/B test in your tool
  5. Calculate the length of your A/B test
  6. Monitoring the A/B test

The change you’re making should be visible, scannable and usable. Do not use “What you see is what you get “ code editor because it’s an experiment. If it works, it works. So, why measure in your own analytics tool? It’s because its the best implementation analytics solution and you have the ultimate control over your experiment.

Results Pillar

  1. The test duration
  2. How to isolate the test population — Does the test have an impact on all the population or a set of few users
  3. Test Goals and how to isolate those users

Once we start the analysis,

  1. Analyse in the analytics tool and not in the test tool
  2. Avoid Sampling
  3. Analyse users and not sessions
  4. Analyse users who have converted and the total conversions
  5. Check if the population of users that have seen the test are about the same per variation.

When we have the analysed report, we need to focus on what information is valuable to present to which group of people. We should get the ability to create an A/B test outcome template that leads to action which means there should have been an impact.



Based on some analytics, it’s calculated that if there’s an increase in the number of users, obviously the conversion might uplift and there would be more noise in the returning users. ( Returning users are people who visit your website for the first time and leave after performing a particular action and then return to the same website later).

Thinking through fingers.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store