Solved – Testing approach Chi Square vs Mann Whitney

Question

I am looking to understand the best approach to determine if students in one school performed better on a quiz than another. In this scenario the quiz has five questions and only whole points are awarded, giving 0, 1, 2, 3, 4, & 5 as the only possible results.

The results:

| score | school_a | school_b |
|-------|----------|----------|
| 0     | 150      | 175      |
| 1     | 50       | 40       |
| 2     | 30       | 30       |
| 3     | 20       | 15       |
| 4     | 5        | 10       |
| 5     | 80       | 90       |

Given the non-normality of the distribution it seems like the Mann-Whitney test would be appropriate, but because scores are integers I am concerned that the ties may cause issues. Should I instead treat the scores as categorical variables and perform a chi-square test instead?

Is there another approach I should be considering instead?

Answer 1

Mann-Whitney (also two-sample Wilcoxon test): Data are

a = rep(0:5, c(150, 50, 30, 20, 5, 80))
b = rep(0:5, c(175, 40, 30, 15, 10, 90))
summary(a)
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.000   0.000   1.000   1.761   4.000   5.000 
 summary(b)
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.000   0.000   1.000   1.764   4.250   5.000 

The means and medians look very nearly the same for the two schools. The formal test has P-value $0.6723 > 0.05$ so the difference is not significant at the 5% level. For samples as large as these, the implementation of the Wilcoxon rank sum test in R, reports no difficulty handling ties.

 wilcox.test(a,b)

        Wilcoxon rank sum test with continuity correction

data:  a and b
W = 61350, p-value = 0.6723
alternative hypothesis: 
   true location shift is not equal to 0

Chisquared test of homogeneity of probabilities: Counts are

f.a = c(150, 50, 30, 20, 5, 80)
f.b = c(175, 40, 30, 15, 10, 90)
MAT = rbind(f.a, f.b);  MAT
    [,1] [,2] [,3] [,4] [,5] [,6]
f.a  150   50   30   20    5   80
f.b  175   40   30   15   10   90

I agree with @Glen_b that the Wilcoxon test is best here, because the issue seems to be which school had the higher scores overall. However, the chi-squared test will test whether the probabilities of getting individual scores 0 through 5 are substantially the same at the two schools. No difference between the two distributions is found.

chisq.test(MAT)

        Pearson's Chi-squared test

data:  MAT
X-squared = 5.1107, df = 5, p-value = 0.4025

Kolmogorov-Smirnov test of of differences in CDFs:

This is a test to see if there is a substantial difference between the empirical CDFs (ECDFs) of the two samples. To make an ECDF, sort the data and plot a stairstep function that increases by $1/n,$ where $n$ is the sample size, at each data value. If there are $k$ tied observations at a point, then jump up by $k/n.$ Here are the ECDFs of for the two schools, blue for a and orange for b.

plot(ecdf(a), col="blue", lwd=2)
lines(ecdf(b), col="orange", lty="dotted", pch="o")

The test statistic $D$ of the K-S test is the maximum vertical distance between the two ECDFs. Again here, no significant difference is found. For your data, this test may be considered an alternative to the chi-squared test.

ks.test(a,b)

        Two-sample Kolmogorov-Smirnov test

data:  a and b
D = 0.03835, p-value = 0.9605
alternative hypothesis: two-sided

Warning message:
In ks.test(a, b) : 
   p-value will be approximate in the presence of ties