I have initially run a 2x2x2 Anova and found some significant interaction between the independent variables. I then ran a ANCOVA where I added a covariate. I am confused with how to interpret the output from the ANCOVA as the significant interactions that I found with the ANOVA are now insignificant in the ANCOVA.
Solved – Why is ANOVA interaction no longer significant after including a covariate
ancovaanovaspss
Related Solutions
This is not a problem. If you know that you want to see if the A-B relationship differs by sex, do a regression that compares the A-B relationship in different sexes. The interaction is the whole point.
As a by-product, this analysis will tell you what the intercept and slopes are in the different sexes, which you can add to your scatterplot.
I am going to use R to demonstrate these concepts, but they can be done in any popular statistics software.
Example Data
The example I like to use is some athletic event. Imagine we have Teams A and B will be facing off against one another. We get people to imagine how happy they will be if Team A wins or if Team B wins. We also ask them if they are a fan of Team A or Team B. Lastly, we get what hand they write with. This gives us four variables: happiness (dependent variable, DV), winning team (first independent variable, IV1), fan of team (IV2), or handedness (IV3).
I simulated data representing this using R:
# generate data ----------------------------------------------------------------
set.seed(1839)
n <- 300
fan <- sample(c("a", "b"), n, TRUE)
win <- sample(c("a", "b"), n, TRUE)
happiness <- rnorm(n, ifelse(fan == win, .5, -.5), 1)
hand <- sample(c("l", "r"), n, TRUE)
The results are programmed to align with what is obvious: The effect of who wins on happiness depends on who the person is a fan of (two-way interaction), but this doesn't depend on handedness (lack of a three-way interaction). People are happy when their team wins, but sad when their team loses.
Main Effect
The main effect can be seen as the effect of one independent variable on the dependent variable, averaging across all other variables. You could also say that it is the effect ignoring or collapsing across all other variables.
To get the main effect of which team wins (A or B) on happiness, we can do a t-test. This ignores all of the other independent variables.
# main effect ------------------------------------------------------------------
t.test(happiness ~ win)
This returns:
Welch Two Sample t-test
data: happiness by win
t = 0.12692, df = 296.16, p-value = 0.8991
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.2346082 0.2669540
sample estimates:
mean in group a mean in group b
-0.08008830 -0.09626121
There is not a significant main effect of which team wins on happiness. This is because it depends on who people are a fan of. We can do that by looking at the interaction.
Fitting Model
I fit a linear model (the same thing as an ANOVA) to look at the main effects, all two-way interactions, and the three-way interaction.
# fit model --------------------------------------------------------------------
model <- lm(happiness ~ fan * win * hand)
summary(model)
This returns a coefficients table that looks like:
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.2060 0.1612 1.278 0.202323
fanb -0.6378 0.2265 -2.815 0.005203 **
winb -0.8181 0.2238 -3.655 0.000304 ***
handr 0.1704 0.2280 0.747 0.455450
fanb:winb 1.5237 0.3275 4.652 4.99e-06 ***
fanb:handr -0.2865 0.3304 -0.867 0.386542
winb:handr -0.1571 0.3147 -0.499 0.617968
fanb:winb:handr 0.7053 0.4620 1.527 0.127927
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
The three-way interaction (fanb:winb:handr) is not significant, but one of the two-way interactions (fanb:winb) is. Now we can "probe" this interaction.
Simple Main Effects
Recall the definition of a main effect:
- The effect of IV1 on the DV, ignoring the effects of all other IVs.
We can change this to the definition of a main effect by changing what comes after the comma:
- The effect of IV1 on the DV, at a specified level of IV2.
These can also be called "conditional" main effects or "conditional average treatment effects" because it looks at the effect of IV1 on the DV given some level of IV2.
So while a main effect says something like, "IV1 does not affect DV," a simple main effect says, "When IV2 = A, then IV1 negatively affects DV." It is that "When IV2 = A" that makes it a simple main effect; it is the main effect of a variable at a given level of another variable (in this case, IV2 = A).
You do not need to fit a different model to examine this. You can estimate it from the model you have. In R, it is as simple as:
# simple main effect -----------------------------------------------------------
library(emmeans)
pairs(emmeans(model, ~ win | fan))
The ~ win | fan is saying that the DV is predicted by (~) the effect of winning, given (|) the level of fandom. This returns:
fan = a:
contrast estimate SE df t.ratio p.value
a - b 0.8966396 0.1573624 292 5.698 <.0001
fan = b:
contrast estimate SE df t.ratio p.value
a - b -0.9797111 0.1691024 292 -5.794 <.0001
So when fan = a, then Team A winning leads to an increase of happiness over Team B by about .9. On the other hand, when fan = b, the effect of Team A winning leads to a decrease of happiness by about .98. The simple main effect part is in specifying the effect of winning at different levels of fandom.
We can also look at it by handedness if we want, but since there was no significant interaction, then it won't make much of a difference:
> pairs(emmeans(model, ~ win | fan + hand))
fan = a, hand = l:
contrast estimate SE df t.ratio p.value
a - b 0.8180739 0.2237993 292 3.655 0.0003
fan = b, hand = l:
contrast estimate SE df t.ratio p.value
a - b -0.7056253 0.2391469 292 -2.951 0.0034
fan = a, hand = r:
contrast estimate SE df t.ratio p.value
a - b 0.9752053 0.2212816 292 4.407 <.0001
fan = b, hand = r:
contrast estimate SE df t.ratio p.value
a - b -1.2537969 0.2391469 292 -5.243 <.0001
Best Answer
First, the difference between significant and non-significant is not necessarily significant. So if after adding a covariate your interaction p-value changed from .04 to .06, this doesn't mean anything substantial other than you've bounced from one side of a binary .05 decision threshold.
However, the other possibility is that the covariate has substantially influenced the analysis. A typical motivation for including a covariate is that you want to control for its effect, and thereby adjust your estimates of main and interaction effects for the covariate.
I recommend that you produce a few plots of the cell mean both controlling and not controlling for the covariate to see what kind of difference inclusion of the covariates has made.