Solved – 3-way ANOVA with a significant 2-way interaction. What are the next steps

anovainteraction

I have performed a mixed 3 way ANOVA: Group (2 levels) x Factor2 (2 levels) x Factor3 (2 levels).

There was no significant 3-way interaction; however, there was a significant 2-way interaction between Group and Factor2.

I am confused by the difference between "main effects" and "simple main effects," and I am not sure what step to take next to understand the 2-way interaction.

Do I need to perform a 2-way ANOVA (Group x Factor2)? Do I need to perform 4 separate tests of some kind?

[UPDATE]

Here is the procedure I used to run the simple main effects:

There was no significant 3 way interaction however there was a significant 2 way interaction of Group*Factor1 and a significant main effect of Factor1.

1) Ran through the same menu option as to explore the 3 way and 2 way interactions. SPSS -> Analyze -> GLM -> Repeated Measures.

2) Clicked 'Paste' in the repeated measures window and adjusted the syntax as follows:

GLM AccF1_L1 Acc_F1_L2 Acc_F2_L1 AccF2_L2 BY Group
/WSFACTOR=Factor1 2 Polynomial Factor2 2 Polynomial
/EMMEANS=TABLES(Factor1*Group) COMPARE(Group).

3) Interpret 'Univariate Tests'
4) If any significant simple main effects go on to interpret pairwise comparisons.

Best Answer

I am going to use R to demonstrate these concepts, but they can be done in any popular statistics software.

Example Data

The example I like to use is some athletic event. Imagine we have Teams A and B will be facing off against one another. We get people to imagine how happy they will be if Team A wins or if Team B wins. We also ask them if they are a fan of Team A or Team B. Lastly, we get what hand they write with. This gives us four variables: happiness (dependent variable, DV), winning team (first independent variable, IV1), fan of team (IV2), or handedness (IV3).

I simulated data representing this using R:

# generate data ----------------------------------------------------------------
set.seed(1839)
n <- 300
fan <- sample(c("a", "b"), n, TRUE)
win <- sample(c("a", "b"), n, TRUE)
happiness <- rnorm(n, ifelse(fan == win, .5, -.5), 1)
hand <- sample(c("l", "r"), n, TRUE)

The results are programmed to align with what is obvious: The effect of who wins on happiness depends on who the person is a fan of (two-way interaction), but this doesn't depend on handedness (lack of a three-way interaction). People are happy when their team wins, but sad when their team loses.

Main Effect

The main effect can be seen as the effect of one independent variable on the dependent variable, averaging across all other variables. You could also say that it is the effect ignoring or collapsing across all other variables.

To get the main effect of which team wins (A or B) on happiness, we can do a t-test. This ignores all of the other independent variables.

# main effect ------------------------------------------------------------------
t.test(happiness ~ win)

This returns:

    Welch Two Sample t-test

data:  happiness by win
t = 0.12692, df = 296.16, p-value = 0.8991
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.2346082  0.2669540
sample estimates:
mean in group a mean in group b 
    -0.08008830     -0.09626121

There is not a significant main effect of which team wins on happiness. This is because it depends on who people are a fan of. We can do that by looking at the interaction.

Fitting Model

I fit a linear model (the same thing as an ANOVA) to look at the main effects, all two-way interactions, and the three-way interaction.

# fit model --------------------------------------------------------------------
model <- lm(happiness ~ fan * win * hand)
summary(model)

This returns a coefficients table that looks like:

Coefficients:
                Estimate Std. Error t value Pr(>|t|)    
(Intercept)       0.2060     0.1612   1.278 0.202323    
fanb             -0.6378     0.2265  -2.815 0.005203 ** 
winb             -0.8181     0.2238  -3.655 0.000304 ***
handr             0.1704     0.2280   0.747 0.455450    
fanb:winb         1.5237     0.3275   4.652 4.99e-06 ***
fanb:handr       -0.2865     0.3304  -0.867 0.386542    
winb:handr       -0.1571     0.3147  -0.499 0.617968    
fanb:winb:handr   0.7053     0.4620   1.527 0.127927    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The three-way interaction (fanb:winb:handr) is not significant, but one of the two-way interactions (fanb:winb) is. Now we can "probe" this interaction.

Simple Main Effects

Recall the definition of a main effect:

The effect of IV1 on the DV, ignoring the effects of all other IVs.

We can change this to the definition of a main effect by changing what comes after the comma:

The effect of IV1 on the DV, at a specified level of IV2.

These can also be called "conditional" main effects or "conditional average treatment effects" because it looks at the effect of IV1 on the DV given some level of IV2.

So while a main effect says something like, "IV1 does not affect DV," a simple main effect says, "When IV2 = A, then IV1 negatively affects DV." It is that "When IV2 = A" that makes it a simple main effect; it is the main effect of a variable at a given level of another variable (in this case, IV2 = A).

You do not need to fit a different model to examine this. You can estimate it from the model you have. In R, it is as simple as:

# simple main effect -----------------------------------------------------------
library(emmeans)
pairs(emmeans(model, ~ win | fan))

The ~ win | fan is saying that the DV is predicted by (~) the effect of winning, given (|) the level of fandom. This returns:

fan = a:
 contrast   estimate        SE  df t.ratio p.value
 a - b     0.8966396 0.1573624 292   5.698  <.0001

fan = b:
 contrast   estimate        SE  df t.ratio p.value
 a - b    -0.9797111 0.1691024 292  -5.794  <.0001

So when fan = a, then Team A winning leads to an increase of happiness over Team B by about .9. On the other hand, when fan = b, the effect of Team A winning leads to a decrease of happiness by about .98. The simple main effect part is in specifying the effect of winning at different levels of fandom.

We can also look at it by handedness if we want, but since there was no significant interaction, then it won't make much of a difference:

> pairs(emmeans(model, ~ win | fan + hand))
fan = a, hand = l:
 contrast   estimate        SE  df t.ratio p.value
 a - b     0.8180739 0.2237993 292   3.655  0.0003

fan = b, hand = l:
 contrast   estimate        SE  df t.ratio p.value
 a - b    -0.7056253 0.2391469 292  -2.951  0.0034

fan = a, hand = r:
 contrast   estimate        SE  df t.ratio p.value
 a - b     0.9752053 0.2212816 292   4.407  <.0001

fan = b, hand = r:
 contrast   estimate        SE  df t.ratio p.value
 a - b    -1.2537969 0.2391469 292  -5.243  <.0001

Related Solutions

Solved – Interpreting a significant 2-way interaction with post-hoc tests

Pooling across the remaining group was correct. You don't need anymore tests. Take choice 1. Your interaction means the effect of difficulty (hard - easy) in controls is different from the effect of difficulty in clinical. Since what it means is exactly what you want to know a post hoc is completely unnecessary.

You might want to see Gelman and Stern (2006) for a lesson in why the path you were thinking of taking with post hocs was faulty logic.

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328-331. doi: 10.1198/000313006X152649

Some general information might help others in the same situation. There seems to be a broadly accepted mistaken belief that there must be a significant simple effect when you have a significant interaction. Not only is this false but there is no guarantee of any significant simple effect in an ANOVA with more than 2 levels (in which case it's just an alternative to a t-test). Interactions are about differences among differences. None of the individual differences needs to be significant, or even close to significant, for a significant interaction. Consider a situation where one effect is +5 and another is -5. The difference between those effects is 10, which is much larger than either simple effect. Therefore, it would be very easy to not have a significant simple effect of 5 but a significant interaction of 10.

With something as simple as a 2x2 designs one should never perform a post hoc on the interaction because the interaction told you all of the necessary information. Even with slightly more complex designs post hoc tests are usually unnecessary, especially when one variable only has 2 levels. People generally do far too many post hoc tests following ANOVAs. In school they tell you to do an ANOVA to avoid multiple comparison problems. Of what benefit is that when, after you perform the ANOVA, you perform a bunch of comparisons? Interpret the ANOVA first.

Solved – 3-way repeated mesures ANOVA

Although this is five years late, I will post as an FYI for anyone else who happens to stumble upon this thread. The first part of the model is properly specified, i.e.

time ~ pres*fact*size

However, the error term is incorrectly specified. Note that in repeated measures ANOVA, "the error term for any within-subject effect is the interaction of that effect with subjects [or whatever entity that is repeatedly measured] (Keppel, G., & Wickens, T. D., 2004, p. 40)."

For this particular example, if using R's aov() function, the error term would be specified as such:

Error(subj/(pres*fact*size))

So in all, we have:

m2 <- aov(time ~ pres*fact*size + Error(subj/(pres*fact*size)), data = data)

WRT the interpretation of a three-way interaction: The implication is that a two-way interaction varied over the levels of a third factor. In other words, there was a two-way interaction that was only manifest at a specific level or levels of a third factor. To get to the heart of the matter, you would need to follow up your omnibus ANOVA with interaction contrasts. To determine what interaction contrasts you should test, it would be helpful to plot your data using a line graph. As it stands, there is not enough information available in your post to provide further guidance.

Edit Perhaps it goes without saying, but this is in the case that you obtain such a result with the model properly specified.