Solved – 2-Sample Proportions z-test vs Fisher’s Exact Test

fishers-exact-testhypothesis testingproportion;

I want to evaluate the difference in proportions in declined applications and exceptions from policy from one quarter of data to the next. My sample sizes are generally not small (several hundred to several thousand), but the proportions may range anywhere from under 1% up to 60% depending on the specific set of data. As the two compared proportions diverge, using either test seems to produce similar p-values. However, if the proportions are relatively close, the z-test and Fisher's test p-values disagree more. Overall, the significance under either test tends to agree. Is one test more appropriate in this scenario based on the my data and any implied assumptions?

My current research seems to suggest that Fisher's Exact Test is always better since it is an exact test and a z-test an asymptotic test. Is this correct? If this is the case, why would anyone with the appropriate computational power ever conduct a z-test for proportions over Fisher's Exact Test?

Best Answer

Fisher's exact test offers a more exact p-value by running every possible scenario with the given data set and figures out the total number of possible successes and total number of possible failures at the given sample size, then it converts those totals into a p-value. Also note that the total possible outcomes goes up by a factorial.

The normal approximation can be computed by a calculator and is much easier to compute as the sample size gets large. The normal approximation also becomes closer to fisher's exact test as the sample size gets large.

Related Solutions

Solved – Fisher’s exact test

The massive 58 amid much lower frequencies signals that any test is just quantifying a major failure of independence. I did this in Stata. The command ret li (short for return list) obliges Stata to show results as exactly as it knows them, but both tests yield P-values that are 0.000 to 3 d.p. It is right to be a little cautious about low expected values (for row 1 here in particular) but the test results are overwhelming.

. tabi 0  2 \ 5 58 \ 4 3 \ 4 3 

            |          col
        row |         1          2 |     Total
 -----------+----------------------+----------
          1 |         0          2 |         2 
          2 |         5         58 |        63 
          3 |         4          3 |         7 
          4 |         4          3 |         7 
 -----------+----------------------+----------
      Total |        13         66 |        79 

      Pearson chi2(3) =  20.5779   Pr = 0.000

. ret li 

scalars:
              r(p) =  .0001288081813192
           r(chi2) =  20.57794057794058
              r(c) =  2
              r(r) =  4
              r(N) =  79

. tabi 0  2 \ 5 58 \ 4 3 \ 4 3 , exact

Enumerating sample-space combinations:
stage 4:  enumerations = 1
stage 3:  enumerations = 3
stage 2:  enumerations = 17
stage 1:  enumerations = 0

             |          col
         row |         1          2 |     Total
  -----------+----------------------+----------
           1 |         0          2 |         2 
           2 |         5         58 |        63 
           3 |         4          3 |         7 
           4 |         4          3 |         7 
  -----------+----------------------+----------
       Total |        13         66 |        79 

       Fisher's exact =                 0.000

. ret li 

scalars:
        r(p_exact) =  .0003124258226793
              r(c) =  2
              r(r) =  4
              r(N) =  79

Solved – Hypothesis testing based on Likert type data

(1) Yes you can, but nothing in either test takes the ordering into account. The ordered scales are reduced to nominal. The tests are applicable, but ignore much of the information you have.

(2) Many researchers would be happy to use rank correlation here. At least it respects the ordering of the categories. Many ties are likely, but that's par for the course. Another test that might be helpful is the Jonckheere-Terpstra test. A more ambitious analysis would be to use an ordinal logit or probit model. I wouldn't put all the weight on a single measure or test of association. Looking at a mosaic plot (spineplot) or two-way bar chart may help you see structure that's not directly expressed by a correlation. For example, on questionnaires some people give almost the same grades regardless of the question.

(3) The contingency table is the same regardless of what tests you apply to it. It's a $5\times5$ table of counts, as you say. It's not clear why you're asking this, not least because producing it is a simple task in any software.

Best Answer

Related Solutions

Solved – Fisher’s exact test

Solved – Hypothesis testing based on Likert type data

Related Question