R – Exact Two Sample Proportions Binomial Test and Interpretation of Strange P-Values

binomial distributionhypothesis testingproportion;rstatistical significance

I am trying to solve the following question:

Player A won 17 out of 25 games while player B won 8 out of 20 – is
there a significant difference between both ratios?

The thing to do in R that comes to mind is the following:

> prop.test(c(17,8),c(25,20),correct=FALSE)

    2-sample test for equality of proportions without continuity correction

data:  c(17, 8) out of c(25, 20)
X-squared = 3.528, df = 1, p-value = 0.06034
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.002016956  0.562016956
sample estimates:
prop 1 prop 2 
  0.68   0.40

So this test says that the difference is not significant at the 95% confidence level.

Because we know that prop.test() is only using an approximation I want to make things more exact by using an exact binomial test – and I do it both ways around:

> binom.test(x=17,n=25,p=8/20)

    Exact binomial test

data:  17 and 25
number of successes = 17, number of trials = 25, p-value = 0.006693
alternative hypothesis: true probability of success is not equal to 0.4
95 percent confidence interval:
 0.4649993 0.8505046
sample estimates:
probability of success 
                  0.68 

> binom.test(x=8,n=20,p=17/25)

    Exact binomial test

data:  8 and 20
number of successes = 8, number of trials = 20, p-value = 0.01377
alternative hypothesis: true probability of success is not equal to 0.68
95 percent confidence interval:
 0.1911901 0.6394574
sample estimates:
probability of success 
                   0.4

Now this is strange, isn't it? The p-values are totally different each time! In both cases now the results are (highly) significant but the p-values seem to jump around rather haphazardly.

My questions

  1. Why are the p-values that different each time?
  2. How to perform an exact two sample proportions binomial test in R correctly?

Best Answer

If you are looking for an 'exact' test for two binomial proportions, I believe you are looking for Fisher's Exact Test. In R it is applied like so:

> fisher.test(matrix(c(17, 25-17, 8, 20-8), ncol=2))
    Fisher's Exact Test for Count Data
data:  matrix(c(17, 25 - 17, 8, 20 - 8), ncol = 2)
p-value = 0.07671
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
  0.7990888 13.0020065
sample estimates:
odds ratio 
  3.101466

The fisher.test function accepts a matrix object of the 'successes' and 'failures' the two binomial proportions. As you can see, however, the two-sided hypothesis is still not significant, sorry to say. However, Fisher's Exact test is typically only applied when a cell count is low (typically this means 5 or less but some say 10), therefore your initial use of prop.test is more appropriate.

Regarding your binom.test calls, you are misunderstanding the call. When you run binom.test(x=17,n=25,p=8/20) you are testing whether proportion is significantly different from a population where the probability of success is 8/20. Likewise with binom.test(x=8,n=20,p=17/25) says the probability of success is 17/25 which is why these p-values differ. Therefore you are not comparing the two proportions at all.

Related Question