Chapter 2- Important topics | AB testing book

What is similarities and difference between AB testing and Causal inference

What is AA test?

What is difference between AA test and AB test

The main difference between an A/A test and an A/B test is that in an A/A test, two identical versions of a page are tested against each other, while in an A/B test, two different versions of a page are tested against each other.

An A/A test is used to verify that your A/B testing tool is working properly. By testing two identical pages, you can be sure that any differences in results are due to the changes you made to the page, and not to any errors in your testing tool.

An A/B test is used to compare two different versions of a page to see which one performs better. This can be used to test changes to the page's design, content, or call to action.

Both A/A tests and A/B tests are important tools for improving the performance of your website or app. However, they are used for different purposes. If you are not sure which type of test you need, it is always best to consult with a web developer or marketing expert.

Why randomization is important in AB testing and in causal inference

If one make observation that people who eat more oat are less likely to have cancer or heart attack. One can observe million people data and make statistical conclusion.

The challenge is people who eat oat may also have other good habits e.g. exercise, walking every day. The population of eating oat vs non eating oat may have many behavior differences.

This is reason we need to identify sample that are similar and randomized in control and treatment group to ensure that treatment is causing the effect.

What is Confidence interval

A confidence interval is a range of values that is likely to contain the true value of a population parameter. In A/B testing, the confidence interval is used to estimate the true difference in conversion rates between two variants.

The confidence interval is calculated by taking the difference in the observed conversion rates for the two variants and adding and subtracting the margin of error. The margin of error is calculated based on the sample size and the confidence level.

The confidence level is the probability that the confidence interval contains the true value of the population parameter. The most common confidence level is 95%, which means that there is a 95% chance that the confidence interval contains the true value of the population parameter.

The confidence interval plays an important role in A/B testing because it allows you to make inferences about the true difference in conversion rates between two variants. For example, if the confidence interval for the difference in conversion rates is 1% to 3%, you can be 95% confident that the true difference in conversion rates is between 1% and 3%.

The confidence interval can also be used to compare the results of multiple A/B tests. For example, if you run two A/B tests with different confidence levels, you can compare the confidence intervals to see which test has a more precise estimate of the true difference in conversion rates.

Overall, the confidence interval is a valuable tool for interpreting the results of A/B tests. By understanding the confidence interval, you can make more informed decisions about which variant to implement and how to measure the success of your A/B tests.

What is Covariate

covariate is a variable that is associated with the outcome variable but is not of primary interest in the study. Covariates are often used in experiment design to control for other variables that could be affecting the outcome.

For example, if you are studying the effect of a new drug on blood pressure, you might want to control for age and weight, since these variables are known to affect blood pressure. You would do this by collecting data on age and weight for all of the participants in your study, and then statistically adjusting for these variables when you analyze your results.

Covariates can also be used to help you understand the mechanism by which your intervention is working. For example, if you are studying the effect of a new educational program on student achievement, you might want to control for prior achievement, since this variable is known to be a strong predictor of future achievement. By controlling for prior achievement, you can get a better understanding of whether the new educational program is actually causing an increase in achievement, or if the increase is simply due to the fact that students who are already high achievers are more likely to participate in the program.

Covariates can be used in a variety of ways in experiment design. They can be used to control for other variables that could be affecting the outcome, to help you understand the mechanism by which your intervention is working, and to make sure that your results are generalizable to the population you are interested in.

Are covariates useful in A/B testing

Yes, covariates can be used in A/B testing. In fact, it is often a good idea to control for covariates when you are running an A/B test. This is because covariates can affect the outcome of your test, even if they are not the main focus of your test.

For example, let's say you are running an A/B test to see which version of a landing page performs better. You might want to control for the time of day that users see the landing page, since this variable could affect how likely they are to convert. You could also control for the user's location, since this could affect their interest in the product or service you are offering.

By controlling for covariates, you can make sure that your results are not due to chance or to other variables that are not related to your test. This will give you more confidence in the results of your A/B test.

How to compute sample size in AB testing

The sample size for an A/B test is the number of people you need to test in order to have a statistically significant result. The formula for calculating sample size is:

Sample size = (Zα/2)2 * p(1-p) / d2

Where:

Zα/2 is the z-score for the desired confidence level (typically 95%).

p is the expected conversion rate of the control group.

d is the minimum detectable difference in conversion rates that you want to be able to detect.

For example, if you expect the control group to have a conversion rate of 2%, and you want to be able to detect a difference in conversion rates of 1% with 95% confidence, then you would need a sample size of 1000 people.

It is important to note that the sample size formula is just a starting point. You may need to adjust the sample size based on other factors, such as the cost of testing, the time it takes to run a test, and the desired level of precision.

Here are some tips for calculating sample size for A/B testing:

Start by estimating the expected conversion rate of the control group. This can be done by looking at historical data or by running a pilot test.

Decide on the minimum detectable difference in conversion rates that you want to be able to detect. This will depend on the business impact of the change you are testing.

Choose a confidence level. The most common confidence level is 95%, but you may want to use a different confidence level depending on the risk you are willing to take.

Use a sample size calculator to calculate the required sample size. There are many sample size calculators available online, such as the one from Optimizely.

Adjust the sample size based on other factors, such as the cost of testing, the time it takes to run a test, and the desired level of precision.

By following these tips, you can calculate the sample size for your A/B test and ensure that you have a statistically significant result.