ABExperiment Types - Different design
How to design ExperimentsClassic A/B Experiment - there is control A and a treatment (varience) B. This is most commonly usd method. Leap Experiment - It is variation of above. Multiple changs/treatment are added together to test whether newer version provide lift. It reduce experimentation time for each change. It has a downside that you do not know which change is contributing how much to lift?
Exploratory Cascade ExperimentIn this method one design multiple variations. Each variation has a single change. The variations build on top of each other and this allows for comparisons across set. (Some teams call each set a flight). It is useful when you have very large traffic and one can use it to try out multiple solutions or simply explore what users like. Exploratory Leap Experiment - In this we explore multiple different combination to determine which change bring benefit. Reverse Experiment - This is similar to above. Here you explictly remove feature to find its effect tect Example : remove price from webpage and see it if it drop conversion.
Intensity Experiment - This type of experiment is useful if hypotheis is that intensity of treatment will impact outcome. There are scenario where a variable has a linear impact on outcome (Example : 1 testimonial, comments on post might be good, 2 even better, 3 even, 100+ is awesome |