Solved – Two-way ANOVA significant, Oneway ANOVA & T-test not signficant

anovartwo-way

Testing for the interaction of Gender on Type (grouping variable, group A & B) for a 'Test' score –

model.lm <- lm(formula= Test~ Type + Gender + Type*Gender,
           data=describeDF3,na.action=na.omit)
Anova(model.lm,type='II',white.adjust='hc3')

White adjust as unequal variance.

Results came up as Type significant and Gender significant but no interaction. I used both a oneway.test ANOVA and t-test to check the Gender effect but in these tests it is coming up as not significant. Why is this?

Is it because the two way test shows whether the variable creates a reduction in the deviance of the lm and the oneway tests look at differences in means?

Any help understanding this would be great!

Rachel

Best Answer

It's impossible to be sure which of (at least) two possible explanations might apply without seeing actual data (or possibly both), but the first thing I'd suggest is that it's probably because if you leave out an important effect the estimated error variance will be inflated, potentially reducing the test-statistic below significance.

(The second obvious possibility relates to potential negative dependence between factors, but I'd start with the inflated error-variance issue)

There's some discussion of both of these in an answer to this related question: If I'm not interested in the interaction, is there any reason to run a Two-Way ANOVA instead of two One-Way ANOVAs?

(the question is rather different, but one of the answers is similar)

Related Solutions

Solved – post hoc test for 3 way anova

Easiest may be to perform regression where you can choose whichever interactions you want:

lm(y ~ temperature + quality + gender, mydata)

or:

lm(y ~ temperature*quality + gender, mydata)

or:

lm(y ~ temperature*quality*gender, mydata)

or:

lm(y ~ temperature*quality*gender - quality*gender, mydata)

ANOVA – Understanding Significant Interaction and Non-Significant Post-Hoc Results

I find it easiest to think about this in the form of a linear regression. Let $Y$ be the continuous outcome, $G$ represent the group (value of 0 for the reference group and 1 for the other), and $S$ represent sex (0 for male, 1 for female). Then your model is:

$$ Y = \beta_0 + \beta_G G + \beta_S S + \beta_{GS}GS$$

with intercept $\beta_0$ the estimated outcome when $G=0$ (reference group) and $S=0$ (male). A "significant" interaction means that you don't accept the null hypothesis of $\beta_{GS}=0$. That might be evaluated with a single F-test comparing 2 models, one with and one without the interaction term. Let's assume that the assumptions of the model are met and that this isn't a "spuriously significant interaction" as @BruceET warns.

Nevertheless, as he also points out, the single F-test of $\beta_{GS}=0$ isn't the same as the test of all 6 pairwise differences among the 4 group/sex combinations that you evidently performed. In particular, the multiple-testing correction makes it harder to rule out that you made no false-positive errors in that set of comparisons. Your Bonferroni correction is particularly (and unnecessarily) strict in that way. For 6 comparisons you need to have p < 0.0083 to establish "signficance" while maintaining a family-wise error rate of p < 0.05.

It's not clear why you needed to do all pairwise comparisons. If you do need to evaluate all pairwise comparisons, then a more powerful test like the Tukey HSD, or at least the Holm modification of the Bonferroni correction, would be better. Even then, as @BruceET put it, "there is no guarantee that even real differences among population means detected by ANOVA will be resolved post hoc."

This type of thing usually happens when the interaction coefficient $\beta_{GS}=0$ is fairly small in magnitude even if "significant" by the usual p < 0.05 criterion. Then you need to apply your knowledge of the subject matter to interpret the results fairly. You might be able to say that the associations of group and sex with outcome aren't strictly additive, but you might not be able to specify just which differences among group/sex combinations are "significantly" different.

Best Answer

Related Solutions

Solved – post hoc test for 3 way anova

ANOVA – Understanding Significant Interaction and Non-Significant Post-Hoc Results

Related Question