Solved – Why does the proportions_ztest function in statsmodels produce different values than the formula for a 1-proportion Z test

pythonstatsmodelsz-test

If I run the following code in Python:

from statsmodels.stats.proportion import proportions_ztest
proportions_ztest(10, 50, 0.5)

the result is (-5.303300858899106, 1.1372725656979709e-07)

However, if I use the formula for a 1-proportion Z test (taken from here):

import math
import scipy.stats

def my_ztest(successes, n, p_0):
    p_hat = successes / n
    z = (p_hat - p_0) / math.sqrt(p_0 * (1 - p_0) / n)
    p = scipy.stats.norm.sf(abs(z)) * 2
    return z, p

my_ztest(10, 50, 0.5)

Both the z-score and the p-value are different: (-4.242640687119285, 2.2090496998585445e-05)

Why does the statsmodels function produce a different value than what is defined by the formula?

Best Answer

proportions_ztest seems to work exactly as documented.

Unfortunately what the documentation says it does is just not what you're expecting it to do.

By default this function uses the sample proportion in calculating the standard error of $p-p_0$. There's an option (via a boolean function argument) to change that default, after which the output should match your own calculation.

I feel the writers of the documentation have let you down: The documentation doesn't make this arguably nonstandard default sufficiently obvious to the naive reader - in fact, it's pretty hard to spot (I found it fairly easily only because I knew to expect something like this).

[The choice itself is reasonable, but failing to make it blatantly clear - e.g. with a warning in bold lettering right at the top of the documentation - amounts to user-vicious behaviour. You don't put such a big "gotcha" into your packages without really good reason, and you make sure people can easily tell when you do.]

Edit: This question has led to a ticket relating to the documentation (see comments below). I applaud the fast action taken -- though the documentation itself may not rapidly change, something productive happened. I am not used to my complaints about documentation (I've made a ton of them here, about various programs) being acted on at all.

Related Question