Sample Size importance in Ab Experimen
Sample Size importance⛔ Myth: A statistically significant double digit conversion rate increase is awesone. 🎉 The bigger conversion uplift, is better for business! 🥳✅ Reality: If lift in conversion is quite big, it is likely to be erronous. 🧨 Check sample size, ypu may be running under power test. . 🧨 💡 Explanation: With low sample size result can appear statistically significant. Sample size is key. Here is example, if 10 users come to website and ab testing split traffic to a and b set. By chance if one user convert on set A an 2 users convert on set B, one may reach to conclusion that there is 200 percent lift. However one should not reach to such conclusion as one extra user can convert by chance. Such test are called under powered test and lead to wrong conclusion. While designing a b expriment it is important to ensure that experiment is not under powered. |