How to calculate sample size based on Conversion Rate, Minimum Detectable Effect, Statistical power and Significance level. I find two very good online tools that calculate it http://www.evanmiller.org/ab-testing/sample-size.html and http://www.adobe-target.com/testcalculator.html (they results are very close). I can't find formla how to calculate this manually.. I tried to calculate myself but got different(much larger) results.
Solved – Calculate sample size based on Conversion Rate, Minimum Detectable Effect, Statistical power and Significance level
ab-testhypothesis testingsample-size
Best Answer
I feel your pain. I blew so many hours yesterday looking at sites like:
And none of them provided a formula, which was disappointing, but the really aggravating part was that their results did not match each other's!
Finally, I found a formula:
$n = (Z_{α/2}+Z_β)^2 * (p_1(1-p_1)+p_2(1-p_2)) / (p_1-p_2)^2$
Then it took me several more hours to convert it to an Excel formula that I could actually use, such as:
=(-NORMSINV(0.05/2)+normsinv(0.8))^2*(0.03*(1-0.03)+0.036*(1-0.036))/(0.03-0.036)^2= 13,911 (Required Observations Per Variation)(This calculator's result matches that of my Excel formula, which is comforting.)
If this answer looks helpful, please upvote.
And see What to do after calculating the required sample size for a Marketing A-B test for more context.