Solved – Why is the Mixed Effects Model (LME) results different from the repeated measures ANCOVA (AOV) results

ancovaanovalme4-nlmemixed modelr

I have been trying to compare the effect of a treatment over time, controlling for a continuous covariate. So two groups (treatment vs control) are given different treatments and compared pre and post. Then I add a continuous covariate into the equation to see if the original effect is still significant. I'm predicting an interaction effect whereby the control group is not different across time, but the treatment group is. The covariate is not of interest, but it is significantly different between groups so there's very likely a confound, so that's why i added it, but that's not why I'm here today.

Basically, the lme command from the nlme package is giving totally different F statistics to aov. I had thought that repeated measures anova was within the general linear model and so were the same as mixed models. My gut says to trust the results of the mixed model, but I don't know how to reconcile the difference. In my original data, it even goes from highly significant to non significant.

My groups are unbalanced but all people went through all time points. Also, my covariate is recorded individually at each time point. If any of those facts sets off any bells.

NOTE: this isn't an off topic question just about R, it's more about why ANOVA/ANCOVA is the same/different from mixed effects models. This probably has something to do with setting contrasts or specifying the model for the SS or something rather than anything to do with R.

Here is some mock data, that is 'like' my original data, but not exactly the same.

    id<-c(1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,14,14)
    group<-c(1,1,1,1,1,1,2,2,2,2,2,2,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2)##group 1 is control
    time<-c(1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2)
    outcome<-c(10,11,12,11,10,11,10,19,10,20,12,20,10,10,11,12,11,10,13,20,10,21,9,19,10,13,10,13)
    covariate<-c(80,82,80,83,79,80,70,80,69,75,69,80,80,84,80,81,80,82,79,85,72,85,72,85,72,73,73,74)

    mydata<-as.data.frame(cbind(id,group,time,outcome,covariate))
    summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id)))
    anova(lme(outcome~factor(group)*factor(time)+covariate, random=~1|id, data=mydata))

Here is my output for ANOVA

    > summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id)))

    Error: id
          Df Sum Sq Mean Sq
    factor(group)  1  4.255   4.255

    Error: Within
                       Df Sum Sq Mean Sq F value   Pr(>F)    
    factor(group)               1  94.98   94.98   57.99 1.33e-07 ***
    factor(time)                1 137.29  137.29   83.81 5.87e-09 ***
    covariate                   1  98.60   98.60   60.20 9.78e-08 ***
    factor(group):factor(time)  1  39.55   39.55   24.15 6.49e-05 ***
    Residuals                  22  36.04    1.64                     
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here is my output for the mixed effects model

    > anova(lme(outcome~factor(group)*factor(time)+covariate, random=~1|id, data=mydata))
                       numDF denDF   F-value p-value
    (Intercept)                    1    12 1249.4127  <.0001
    factor(group)                  1    12   23.7548   4e-04
    factor(time)                   1    11  115.8036  <.0001
    covariate                      1    11   75.0274  <.0001
    factor(group):factor(time)     1    11   28.0040   3e-04

As you can see totally different. Any help would be greatly appreciated.

Best Answer

There are a few reasons as to why an ANCOVA might give you different results from an a linear mixed-effects model (LMM). ANOVAs/ANCOVAs require a very strict set of assumptions to be met for them to be appropriate for ones data, and these assumptions are rarely met, while LMMs are typically quite flexible. Due to these strict assumptions ANOVA/ANCOVAs aren't commonly used, at least not correctly.

I suspect that the main reason the two are giving very different results is that ANOVA's and ANCOVAs typically fall apart when the group sample sizes are different. Repeated measures ANOVA/ANCOVA can't cope with the fact that there are different numbers of individuals in each group within the time points measured. Mixed models can cope with this by altering the estimation for each Time × Group combinations variability.

This article goes through all the limitations of ANOVAs/ANCOVAs quite succinctly if you want to learn more about them. Let me know if you want any more clarification though!

TL;DR: ANOVAs can't cope with unbalanced groups, while LMMs can. Don't use ANOVAs/ANCOVAs.

Related Question