Chapter 3- Randomization and challenges | AB testing book

Randomization sound simple but attrition, spillover, mediation impact experiements. Undertand these before you start experiment.

AB testing experiments can be susceptible to attrition challenges if the groups being tested are not equally matched in terms of size, demographics, or other important factors. This can lead to results that are not indicative of the true effect of the treatments being tested.

Attrition

In a controlled experiment, attrition is the loss of participants from the study due to death, illness, relocation, or other reasons.

Attrition and bias

Attrition can lead to bias if it is not random.

Types of attrition

There are three types of attrition:

Examples of attrition in abtesting

A company was testing two different versions of a landing page for a new product. One version of the page had a longer form that required more information from visitors, while the other version had a shorter form that required less information. The company found that the shorter form had a higher conversion rate, but they also noticed that more visitors dropped out of the test before completing the form. When they investigated further, they found that the visitors who dropped out were more likely to be people who were not interested in the product. This suggests that the attrition bias was caused by the fact that the shorter form was more likely to be seen by people who were not interested in the product in the first place.

A company was testing two different versions of an email campaign. One version of the email had a more persuasive headline, while the other version had a less persuasive headline. The company found that the more persuasive headline had a higher open rate, but they also noticed that more people unsubscribed from the email list after receiving the more persuasive headline. When they investigated further, they found that the people who unsubscribed were more likely to be people who were already on the email list and who were not interested in the product being promoted. This suggests that the attrition bias was caused by the fact that the more persuasive headline was more likely to be seen by people who were already on the email list and who were not interested in the product in the first place.

Examples - spillover in AB testing

Here are a few real examples of A/B testing in which spillover played a role:

A company was testing two different versions of a landing page for a new product. One version of the page had a social media sharing button, while the other version did not. The company found that the version with the social media sharing button had a higher conversion rate, but they also noticed that more people who saw the page with the social media sharing button shared it on social media. This suggests that the spillover effect was caused by the fact that people who saw the page with the social media sharing button were more likely to be interested in the product and to share it with their friends.

A company was testing two different versions of an email campaign. One version of the email had a call to action that asked people to visit the company's website, while the other version did not. The company found that the version with the call to action had a higher click-through rate, but they also noticed that more people who saw the email with the call to action visited the company's website. This suggests that the spillover effect was caused by the fact that people who saw the email with the call to action were more likely to be interested in the company's products or services and to visit the company's website.

A company was testing two different versions of a checkout page. One version of the checkout page had a loyalty program sign-up form, while the other version did not. The company found that the version with the loyalty program sign-up form had a higher conversion rate, but they also noticed that more people who saw the page with the loyalty program sign-up form signed up for the loyalty program. This suggests that the spillover effect was caused by the fact that people who saw the page with the loyalty program sign-up form were more likely to be interested in the company's products or services and to sign up for the loyalty program.

These are just a few examples of how spillover can affect A/B testing results. It is important to be aware of this bias and to take steps to mitigate it. Some of the ways to mitigate spillover bias include:

Using a control group that is not exposed to the treatment.

Using a counterfactual analysis to estimate the effect of the treatment.

Using a Bayesian approach to A/B testing.

By taking these steps, you can increase the validity and reliability of your A/B testing results and make better decisions for your business.

Mediation

In a controlled experiment, mediation is a statistical technique that is used to determine the extent to which the effect of an independent variable (IV) on a dependent variable (DV) is mediated by an intervening variable (M). Mediation is applicable in A/B testing, as it can be used to determine the extent to which the effect of a treatment on an outcome is mediated by a mediating variable.

For example, a company might want to test the effect of a new marketing campaign on sales. The company could randomly assign customers to two groups: one group would be exposed to the new marketing campaign, and the other group would not be exposed to the new marketing campaign. The company could then measure sales for both groups.

If the company finds that the new marketing campaign has a positive effect on sales, it might want to know why the new marketing campaign is effective. Mediation analysis could be used to determine the extent to which the effect of the new marketing campaign on sales is mediated by a mediating variable, such as brand awareness or customer interest.

If the company finds that brand awareness is a mediator, this would mean that the new marketing campaign is effective because it increases brand awareness, and that increased brand awareness in turn leads to increased sales.

Mediation analysis can be a useful tool for understanding the causal effects of interventions. It can be used to determine the extent to which the effect of an intervention on an outcome is mediated by a mediating variable. This information can be used to improve the design and implementation of interventions.

Heterogeneity, covariates and design in controlled experiment

Heterogeneity can be explored by looking at the variability in the results of the controlled experiment. This can be done by looking at the standard deviation, range, or other measures of variability. Covariates can be explored by looking at the relationship between the independent variable and the dependent variable. This can be done by looking at the correlation between the two variables. Design can be explored by looking at the way the experiment is designed. This can be done by looking at the way the independent and dependent variables are assigned to the different groups.

Hetrogeneity in controlled experiments

Heterogeneity is the variability in a population or sample. In a controlled experiment, heterogeneity can be a source of confounding, meaning that it can make it difficult to determine the true effect of the independent variable.

There are a number of ways to explore heterogeneity in a controlled experiment. One way is to use a factorial design, which is a design that includes multiple independent variables. This allows the researcher to control for the effects of heterogeneity by looking at the interaction between the independent variables.

Another way to explore heterogeneity is to use a stratified random sample. This is a sample that is divided into groups based on the levels of the independent variables. The researcher then randomly selects participants from each group. This helps to ensure that the sample is representative of the population and that the effects of heterogeneity are minimized.

Covariates are variables that are correlated with the independent variable but are not of primary interest in the study. In a controlled experiment, covariates can be a source of confounding, meaning that they can make it difficult to determine the true effect of the independent variable.

There are a number of ways to deal with covariates in a controlled experiment. One way is to control for them statistically. This can be done by using a regression analysis to adjust for the effects of the covariates.

Another way to deal with covariates is to remove them from the study. This can be done by matching participants on the covariates or by using a statistical technique called propensity score matching.

The design of a controlled experiment can also affect the extent to which heterogeneity and covariates are a problem. A well-designed experiment will minimize the effects of heterogeneity and covariates by using a factorial design, a stratified random sample, and statistical controls.

By carefully considering the issues of heterogeneity, covariates, and design, researchers can increase the confidence that the results of their controlled experiments are valid.