Learn about AB testing | AB Experiment

A/B test is method to run controlled experiment. There are scenarios where A/B testing is most suitable and there are scenario where A/B testing does not make sense.

When A/B test/ experimemt is right strategy

A/B experiment are good for answering question when you need to learn from user. A/B test is a user research tool. It let you determine what you cant determine in meeting or in lab exercise. It is

  • Display smaller title or bigger
  • Should we display price upfront or show later when user have shown interest
  • Should we display button in blue color or red
  • should we display medium image or large image
  • Should we display movie name or movie_name + actor in netflix kind of platform
  • Should we add recommendation


    • A/B test are controlled experiment. To run successful a-b test we need to have sufficient usr base/traffic, ability to randomly select users into a and b buckets and give different treatment and experience.
    /A/B test manipulate one variable at one time. these are considered univariate test. Once company gain expertise they can run few experiment as multi variate test too.
    These are not perfect for all situation. Example
  • What should be price in market X

    • sometime running an ab test may be too costly . Example
  • Will user stop using our product if we show 3 ads

    • Sometime running an ab test is not an option Example
  • how user body will react to vaccine dose increase by 50%

    • Sometime running an ab test is not an option Example
  • how user body will react in car crash by adding extra air bags


    • AB test should be conducted user in ming. Areas where you wish to learn user preference and impact of treatment is not dangerous/not unethical, AB testing can be considered.
    Many e-commerce companies have a policy to run ab test before they change their site/recommendation. Example Google, Facebook, Microsoft, Netflix run extensive ab testing before they introduce user impacting changes.

    Who get involve in A/B experiment

    If you want to start ab experiment in your company - start with

    Data Scientist or Machine Learning Team


    They can provide initial pointers e.g. where it can be applicable, have applied, what worked, what did not worked. They can also describe whether company has enough traffic sample-size etc.

    Marketing team

    In many companies marketing team own website and they have formal and informal method to determine what factors are improving engagement on website. In these scenario check with marketing team whether they have considered A/B testing

    Sales Team

    Sales team focus on sale from website. Many times they have considered or done ab testing.

    In large company both marketing and sales team influence website. There is famous story about Microsoft AB testing in which change done by marketing team improve marketing result and but nagatively impact sales revenue.

    AB testing in built platform or vendor platform

    If your company has in built experimentation platform or have bought a platform form external vendor - you should look into these.You should check what experiments are run/in-progress and history of execution

    Overhaul or complete redesign

    Sometime companies do complete redesign of their website, brand. Brining many changes together may not be suitable for A/B testing and they may rely on market research. Once things are stabilized incremental changes can be introduced with AB testing.

    Planning A/B tests

    You need to ensure you met prereq for ab test. You need to plan how you will run ab test. what assumptions you have, what data you will collect, how will you evaluate result and what action you will take if you are able to validate/invalidate hypothesis.
    You need to have at least these basic capabilities

  • Randomization
  • Ability to split traffic into bucket A and bucket B
  • Ability to collect data
  • Determined sample size and power
  • decide how much percentage of traffic you will give to flight A and flight B
  • Determine how long you need to run test for making conclusion

    • Ideally you need to maintain consistency e.g.
  • Experiment: your experiment have button in color "blue" (A) and "red" (B).
  • if user "M" come to your website yesterday and today and
    • same user should be either in bucket A or in bucket B for both days.

    Execute A/B test

  • Generate hypothesis
  • Check traffic and data needed
  • Review Sample size and power of test are correct
  • Test you can generate variate for treatment B in reliable manner
  • Test you can generate percentage of traffic you will give to flight A and flight B
  • Start test
  • Keep checking that traffic is going to both flight
  • Keep checking that there is nothing drastic happening in flight B, otherwise you may have to consider stopping test
  • Regularly review the result with team
    •

    Topic

  • Home
  • 1 overview
  • 2 hypothesis testing
  • 3 covariates
  • 4 non compliance
  • 5 attrition
  • 6 spillover
  • 7 heterogeneous treatment effects
  • 8 mediation
  • Ab testing on cloud
  • Methods
  • Slides and blog

    Read more