Solved – calculating necessary sample size

hypothesis testingsample-sizet-test

I am trying to use python and online tools to calculate the accurate sample size. However, each way I use I get a different result.

This is the data I have from a previous test

Control Group

  • Sent:140000
  • Converted: 6000
  • Conversion Rate: 0.0429

Test/Treatment:

  • Sent:350000
  • Converted:19000
  • Conversion Rate: 0.0543

If I use this calculator http://www.evanmiller.org/ab-testing/sample-size.html
with

  • Baseline conversion rate 4.29
  • minimum detectable effect 2% ( I don't get results for 20%)
  • 1 – Beta: 80%
  • alpha: 5%
  • RESULT: 1,713

If I use this calculator https://www.optimizely.com/resources/sample-size-calculator/?conversion=4.29&effect=20&significance=95
with

  • Baseline conversion rate 4.29
  • minimum detectable effect: 20%
  • statistical significance 95%
  • RESULT: 8,300

same calculator

  • Baseline conversion rate 4.29
  • minimum detectable effect 20% (if I enter 2%I get a sample size
    required of 1,200,000)
  • statistical significance 90%
  • RESULT: 7,900

Finally, when I use python (code from here https://stackoverflow.com/questions/15204070/is-there-a-python-scipy-function-to-determine-parameters-needed-to-obtain-a-ta) 

from scipy.stats import norm, zscore

def sample_power_probtest(p1, p2, power=0.8, sig=0.05):
    z = norm.isf([sig/2]) #two-sided t test
    zp = -1 * norm.isf([power]) 
    d = (p1-p2)
    s =2*((p1+p2) /2)*(1-((p1+p2) /2))
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

def sample_power_difftest(d, s, power=0.8, sig=0.05):
    z = norm.isf([sig/2])
    zp = -1 * norm.isf([power])
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

if __name__ == '__main__':
    n = sample_power_probtest(0.0429, 0.0543, power=0.8, sig=0.05)
    print n

and I get RESULT: 5585

Best Answer

For quick calculation, one can use following simplified formula: 

sample size = 16 * p * (100-p) / (d ^ 2)

where p = baseline proportion in percent

and d = absolute percent difference

If p=4.29 and d=5.43-4.29=1.14

sample size = 5055

Which is very close to accurate calculations using proper formulae.

Also, if you feed above p and d at https://www.evanmiller.org/ab-testing/sample-size.html

you get sample size of 5,142 which is also close and consistent.

On https://www.optimizely.com/sample-size-calculator/?conversion=4.29&effect=26.6&significance=95 you have feed relative percent difference, i.e. (1.14/4.29)*100 = 26.6%. With these values you get sample size of 4500, which is not close for reasons unclear to me.

Related Question