Specifically, two group ANOVA vs unpaired t-test. My understanding was that those would always give the same p-value, but someone just told me that there could be rare cases where the ANOVA and the t-test would lead to different p-values. My questions: (a) is this true? and if yes, (b) under what conditions would this be more likely to occur? Thanks.
Solved – Can ANOVA and t-test give different results
anovat-test
Related Solutions
Using ANOVA in R does not produce anything different from using ANOVA in another program, and with two groups the results will be equivalent to an equal variance t-test. The t-test is known to be robust to deviations from normality, though with unequal variances a Welch's t-test is probably preferrable.
In the special case of a score based on the number of correct answers on a multiple choice test, the distribution of the score is probably an overdispersed binomial. In that case the "correct" analysis might be a GLM model with a quasi-binomial distribution. Of course, the results might be quite similar to the unequal variance t-test.
Here is a simple simulation based example with 20 questions, and unequal variances. Welch's t-test gives a result much closer to the overdispersed binomial regression.
set.seed(3413)
#generate first sample
p1 <- 1/(1+exp(-1+rnorm(10,sd=1)))
x1 <- rbinom(10, size=20, p=p1)
#generate second sample
p2 <- 1/(1+exp(-3+rnorm(10, sd=1)))
x2 <- rbinom(10, size=20, p=p2)
#combine two sets
x <- c(x1,x2)
g <- gl(2,10)
#summaries:
tapply(x, g, mean)
1 2
12.6 19.2
tapply(x, g, sd)
1 2
3.921451 1.032796
#t-test:
t.test(x ~ g, var.equal=TRUE)
Two Sample t-test
data: x by g
t = -5.1468, df = 18, p-value = 6.765e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-9.294136 -3.905864
sample estimates:
mean in group 1 mean in group 2
12.6 19.2
#without equal variances:
t.test(x ~ g, var.equal=FALSE)
Welch Two Sample t-test
data: x by g
t = -5.1468, df = 10.243, p-value = 0.0004016
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-9.448128 -3.751872
sample estimates:
mean in group 1 mean in group 2
12.6 19.2
#overdispersed binomial regression:
summary(glm(cbind(x, 20-x) ~ g, family="quasibinomial") )
Call:
glm(formula = cbind(x, 20 - x) ~ g, family = "quasibinomial")
Deviance Residuals:
Min 1Q Median 3Q Max
-2.5340 -0.8386 -0.2199 1.2778 2.7581
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.5322 0.2242 2.374 0.028946 *
g2 2.6458 0.5962 4.438 0.000318 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for quasibinomial family taken to be 2.343713)
Null deviance: 120.242 on 19 degrees of freedom
Residual deviance: 45.197 on 18 degrees of freedom
AIC: NA
Number of Fisher Scoring iterations: 5
In general, you wouldn't necessarily expect one way ANOVA and the Kruskal-Wallis to be similar, sometimes they can give quite different p-values. See here for a little partial motivation for why you might expect a difference. [When samples are reasonably normal-looking and with means not too many standard errors apart, they often tend to give similar p-values. Outside that, they frequently don't.]
However, in this case the reason is more prosaic: Your Kruskal-Wallis p-value is wrong.
Here's a summary of results in R (details below).
p-value
Welch t-test: 0.001287
Equal-var. t-test: 8.552e-05
One way anova: 8.55e-05
Wilcoxon test: 0.004847
Kruskal-Wallis: 0.003761
(Neither of the last two p-values are exact; if they were, you'd get the same p-value for the two-group comparison.)
Your problem is you're treating the second group's data as a factor (see the end of this answer).
Here's what I get in R with your data:
frh <- data.frame(group1 = c(103.56, 103.32, 103.32, 104.27, 103.56, 103.8),
group2 = c( 97.16, 97.16, 96.69, 98.58, 90.76, 97.64))
# strip chart:
# Welch t-test:
> with(frh,t.test(group1,group2))
Welch Two Sample t-test
data: group1 and group2
t = 6.3316, df = 5.163, p-value = 0.001287
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
4.368147 10.245186
sample estimates:
mean of x mean of y
103.63833 96.33167
$\,$
# equal-variance t-test:
> with(frh,t.test(group1,group2,var.equal=TRUE))
Two Sample t-test
data: group1 and group2
t = 6.3316, df = 10, p-value = 8.552e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
4.735411 9.877922
sample estimates:
mean of x mean of y
103.63833 96.33167
$\,$
#one way anova:
summary(aov(values~ind,stack(frh)))
Df Sum Sq Mean Sq F value Pr(>F)
ind 1 160.16 160.2 40.09 8.55e-05 ***
Residuals 10 39.95 4.0
$\,$
# Wilcoxon-Mann-Whitney:
> with(frh,wilcox.test(group1,group2))
Wilcoxon rank sum test with continuity correction
data: group1 and group2
W = 36, p-value = 0.004847
alternative hypothesis: true location shift is not equal to 0
Warning message:
In wilcox.test.default(group1, group2) :
cannot compute exact p-value with ties
$\,$
# Kruskal-Wallis test:
> kruskal.test(frh)
Kruskal-Wallis rank sum test
data: frh
Kruskal-Wallis chi-squared = 8.3958, df = 1, p-value = 0.003761
Those are all about as consistent with each other as I would expect on that data.
Now, here's how to get what you got for the Kruskal-Wallis:
with(frh,kruskal.test(group1,group2))
Kruskal-Wallis rank sum test
data: group1 and group2
Kruskal-Wallis chi-squared = 4.3939, df = 4, p-value = 0.3553
The problem is, if you're getting this, you're using it wrong. That's not how the function works - group2 is being treated as a factor defining different groups for data in group1.
So the main reason the Kruskal Wallis isn't giving you a roughly similar p-value to ANOVA is you didn't call it correctly.
Best Answer
The two should give the same results in all cases. ANOVA is a generalization of t-test.