Solved – a valid post-hoc analysis for a three-way repeated measures ANOVA

anovainteractionpost-hocrrepeated measures

I've performed a three-way repeated measures ANOVA; what post-hoc analyses are valid?

This is a fully balanced design (2x2x2) with one of the factors having a within-subjects repeated measure. I'm aware of multivariate approaches to repeated measures ANOVA in R, but my first instinct is to proceed with a simple aov() style of ANOVA:

aov.repeated <- aov(DV ~ IV1 * IV2 * Time + Error(Subject/Time), data=data)

DV = response variable

IV1 = independent variable 1 (2 levels, A or B)

IV2 = independent variable 2 (2 levels, Yes or No)

IV3 = Time (2 levels, Before or After)

Subject = Subject ID (40 total subjects, 20 for each level of IV1: nA = 20, nB = 20)

summary(aov.repeated)

    Error: Subject
          Df Sum Sq Mean Sq F value   Pr(>F)   
IV1       1   5969  5968.5  4.1302 0.049553 * 
IV2       1   3445  3445.3  2.3842 0.131318   
IV1:IV2   1  11400 11400.3  7.8890 0.007987 **
Residuals 36  52023  1445.1                    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Error: Subject:Time
               Df Sum Sq Mean Sq F value   Pr(>F)   
Time            1    149   148.5  0.1489 0.701906   
IV1:Time        1    865   864.6  0.8666 0.358103   
IV2:Time        1  10013 10012.8 10.0357 0.003125 **
IV1:IV2:Time    1    852   851.5  0.8535 0.361728   
Residuals      36  35918   997.7                    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Alternatively, I was thinking about using the nlme package for a lme style ANOVA:

aov.repeated2 <- lme(DV ~ IV1 * IV2 * Time, random = ~1|Subject/Time, data=data)
summary(aov.repeated2)

Fixed effects: DV ~ IV1 * IV2 * Time 
                                Value Std.Error DF   t-value p-value
(Intercept)                      99.2  11.05173 36  8.975972  0.0000
IV1                              19.7  15.62950 36  1.260437  0.2156
IV2                              65.9  15.62950 36  4.216385  0.0002 ***
Time                             38.2  14.12603 36  2.704228  0.0104 *
IV1:IV2                         -60.8  22.10346 36 -2.750701  0.0092 **
IV1:Time                        -26.2  19.97722 36 -1.311494  0.1980
IV2:Time                        -57.8  19.97722 36 -2.893295  0.0064 **
IV1:IV2:Time                     26.1  28.25206 36  0.923826  0.3617

My first instinct post-hoc of significant 2-way interactions with Tukey contrasts using glht() from multcomp package:

data$IV1IV2int <- interaction(data$IV1, data$IV2)
data$IV2Timeint <- interaction(data$IV2, data$Time)

aov.IV1IV2int <- lme(DV ~ IV1IV2int, random = ~1|Subject/Time, data=data)
aov.IV2Timeint <- lme(DV ~ IV2Timeint, random = ~1|Subject/Time, data=data)

IV1IV2int.posthoc <- summary(glht(aov.IV1IV2int, linfct = mcp(IV1IV2int = "Tukey")))
IV2Timeint.posthoc <- summary(glht(aov.IV2Timeint, linfct = mcp(IV2Timeint = "Tukey")))

IV1IV2int.posthoc
#A.Yes - B.Yes == 0        0.94684   
#B.No - B.Yes == 0         0.01095 * 
#A.No - B.Yes == 0         0.98587    I don't care about this
#B.No - A.Yes == 0         0.05574 .  I don't care about this
#A.No - A.Yes == 0         0.80785   
#A.No - B.No == 0          0.00346 **

IV2Timeint.posthoc 
#No.After - Yes.After == 0           0.0142 *
#Yes.Before - Yes.After == 0         0.0558 .
#No.Before - Yes.After == 0          0.5358   I don't care about this
#Yes.Before - No.After == 0          0.8144   I don't care about this
#No.Before - No.After == 0           0.1941  
#No.Before - Yes.Before == 0         0.8616

The main problem I see with these post-hoc analyses are some comparisons that aren't useful for my hypotheses.

Any suggestions for an appropriate post-hoc analysis are greatly appreciated, thanks.

Edit: Relevant question and answer that points toward testing manual contrast matrices

Best Answer

I think statisticians will tell you that there is always a problem with any post hoc analysis because seeing the data may influence what you look at and you could be biased becuase you are hunting for significant results. The FDA in clinical trial studies requires that the statistical plan be completely spelled out in the protocol. in a linear model you certainly could prespecify the contrasts that you would like to look at in the event that the ANOVA or ANCOVA finds an overall difference. Such prespecified contrasts would be fine to look at as long as the usual treatment for multiplicity is also part of it.

Related Question