I am talking about a hypergeometric test but the logic probably applies as well to the binomial test (and other similar tests).
Basic Scenario
I have a population of $n+m$ balls in an urn. There are $m$ white balls and $n$ black balls. I draw a sample of $k$ balls without replacement. I get $q$ white balls and $k-q$ black balls.
-
Null hypothesis: White and black balls have the same probability of being sampled.
-
Alternative hypothesis: White and black balls DO NOT have the same probability of being sampled.
One-tailed hypergeometric test
As there is no replacement, the probability of obtaining $q$ white balls is given by a hypergeometric distribution. I can calculate the probability of getting $k$ or less than $k$ balls in my sample by adding all the probability over the range k to 0. In other words, I can compute $\sum_{i=0}^q P(i)$, where $P(i)$ is the probability of of drawing $i$ balls given by the hypergeometric function. As I am using R, here is the R code for this calculation phyper(q=q, m=m, n=n, k=k).
Similarly, I could compute the probability of obtaining $q$ or more balls: $\sum_{i=q}^k P(i)$. In R it gives phyper(q=q-1, m=m, n=n, k=k, lower.tail = FALSE)
My question is…
How can I get a p.value for a two-tailed hypergeometric test?
From wiki I read about two-tailed binomial test:
There are two methods to define the two tailed p-value. One method is to sum the probability that the total deviation in numbers of events in either direction from the expected value is either more than or less than the expected value. [..] The second method involves computing the probability that the deviation from the expected value is as unlikely or more unlikely than the observed value, i.e. from a comparison of the probability density functions
I think this is very related to what I need but I don't quite get it!
Best Answer
I ended up considering the mass of probability on the shortest tail and multiplied this probability by 2 to account for the fact that it is a two-tailed test.
I have no reference that this is the best method but it felt quite intuitive to me!