Solved – Why do anova(type=’marginal’) and anova(type=’III’) yield different results on lmer() models

anovalme4-nlme

When analysing mixed-effects data using lmer() I find that using anova(type='marginal') and anova(type='III') give different results. Why the discrepancy?

The results from anova(type='marginal') are identical to those I get from using car:Anova(type='III') on the same model and to using both anova(type='marginal') and car:Anova(type='III') on the same data fitted using lme().

library(lme4)
library(nlme)
library(car)

# Some data
group<-as.factor(c(rep('A',32),rep('B',32)))
time<-as.factor(rep(c(rep('t1',16),rep('t2',16)),2))
cond<-as.factor(rep(c(rep('c1',8),rep('c2',8)),4))
subject<-as.factor(rep(c(rep(1:8,2),rep(9:16,2)),2))
set.seed(1)
dv<-c(rnorm(16, mean=3),rnorm(16, mean=1),rnorm(16, mean=0),rnorm(16, mean=3))
data<-data.frame(dv,group,time,cond, subject)

# Model using both lme() and lmer()
model_lme<-lme(dv~group*time*cond, random=~1|subject, data = data, method='ML')
model_lmer<-lmer(dv~group*time*cond+ (1|subject), data = data, REML=FALSE)

# Investigate output of same model with lme() og lmer()
'lme() anova(type=marginal):'
anova(model_lme, type='marginal')
'lme() car::Anova(type=III):'
Anova(model_lme,type='III')
'lmer() anova(type=III):'
anova(model_lmer, type='III')
'lmer() anova(type=marginal):'
anova(model_lmer, type='marginal')
'lmer() car::Anova(type=III):'
Anova(model_lmer,type='III')

Best Answer

This is admittedly confusing, but there are a bunch of differences/limitations between anova() and car::Anova() and between lme and lmer fits. tl;dr probably best to use car::Anova() for consistent results across model types.

anova on lme: allows type="sequential" or type="marginal" (only). type="marginal" should be closest to type-3. Returns F-statistics with denominator degrees of freedom (ddf) calculated by "inner-outer" method (group/parameter counting).
Anova on lme: allows type="II" or type="III". Returns chi-square statistics (i.e. denominator df) only.
anova on lmer: returns sequential F statistics, with no p-values, unless you have the lmerTest package loaded, in which case you get a choice of type II vs III and a choice of ddf calculations
Anova on lmer: if you fit with REML=TRUE you can specify test="F" and get a choice of ddf calculations.

So reviewing the tests you did:

anova(model_lme, type='marginal'): type-III/marginal F tests, inner-outer ddf
Anova(model_lme,type='III'): similar, but returns chi-square statistics/p-values instead, so the p-values are slightly anti-conservative (e.g. p-value for time effect is 0.0012 for F(42) and $1.49 \times 10^{-5}$ for chi-square)
anova(model_lmer, type='III'): the type argument is ignored, so you get sequential/type-I F values (similar but ??? not identical to anova(model_lme, type='sequential')
anova(model_lmer, type='marginal'): ditto
Anova(model_lmer,type='III'): type-3, but chi-square: identical to Anova(model_lme, type='III')

If you refit both models with REML, Anova(model_lmer,type="III",test="F") and anova(model_lme,type="marginal") give similar results.

Related Solutions

ANOVA and MANOVA – How to Interpret Type I, Type II, and Type III ANOVA and MANOVA?

What you are calling type II SS, I would call type III SS. Lets imagine that there are just two factors A and B (and we'll throw in the A*B interaction later to distinguish type II SS). Further, lets imagine that there are different $n$s in the four cells (e.g., $n_{11}$=11, $n_{12}$=9, $n_{21}$=9, and $n_{22}$=11). Now your two factors are correlated with each other. (Try this yourself, make 2 columns of 1's and 0's and correlate them, $r=.1$; n.b. it doesn't matter if $r$ is 'significant', this is the whole population that you care about). The problem with your factors being correlated is that there are sums of squares that are associated with both A and B. When computing an ANOVA (or any other linear regression), we want to partition the sums of squares. A partition puts all sums of squares into one and only one of several subsets. (For example, we might want to divide the SS up into A, B and error.) However, since your factors (still only A and B here) are not orthogonal there is no unique partition of these SS. In fact, there can be very many partitions, and if you are willing to slice your SS up into fractions (e.g., "I'll put .5 into this bin and .5 into that one"), there are infinite partitions. A way to visualize this is to imagine the MasterCard symbol: The rectangle represents the total SS, and each of the circles represents the SS that are attributable to that factor, but notice the overlap between the circles in the center, those SS could be given to either circle.

enter image description here

The question is: How are we to choose the 'right' partition out of all of these possibilities? Let's bring the interaction back in and discuss some possibilities:

Type I SS:

SS(A)
SS(B|A)
SS(A*B|A,B)

Type II SS:

SS(A|B)
SS(B|A)
SS(A*B|A,B)

Type III SS:

SS(A|B,A*B)
SS(B|A,A*B)
SS(A*B|A,B)

Notice how these different possibilities work. Only type I SS actually uses those SS in the overlapping portion between the circles in the MasterCard symbol. That is, the SS that could be attributed to either A or B, are actually attributed to one of them when you use type I SS (specifically, the one you entered into the model first). In both of the other approaches, the overlapping SS are not used at all. Thus, type I SS gives to A all the SS attributable to A (including those that could also have been attributed elsewhere), then gives to B all of the remaining SS that are attributable to B, then gives to the A*B interaction all of the remaining SS that are attributable to A*B, and leaves the left-overs that couldn't be attributed to anything to the error term.

Type III SS only gives A those SS that are uniquely attributable to A, likewise it only gives to B and the interaction those SS that are uniquely attributable to them. The error term only gets those SS that couldn't be attributed to any of the factors. Thus, those 'ambiguous' SS that could be attributed to 2 or more possibilities are not used. If you sum the type III SS in an ANOVA table, you will notice that they do not equal the total SS. In other words, this analysis must be wrong, but errs in a kind of epistemically conservative way. Many statisticians find this approach egregious, however government funding agencies (I believe the FDA) requires their use.

The type II approach is intended to capture what might be worthwhile about the idea behind type III, but mitigate against its excesses. Specifically, it only adjusts the SS for A and B for each other, not the interaction. However, in practice type II SS is essentially never used. You would need to know about all of this and be savvy enough with your software to get these estimates, and the analysts who are typically think this is bunk.

There are more types of SS (I believe IV and V). They were suggested in the late 60's to deal with certain situations, but it was later shown that they do not do what was thought. Thus, at this point they are just a historical footnote.

As for what questions these are answering, you basically have that right already in your question:

Estimates using type I SS tell you how much of the variability in Y can be explained by A, how much of the residual variability can be explained by B, how much of the remaining residual variability can be explained by the interaction, and so on, in order.
Estimates based on type III SS tell you how much of the residual variability in Y can be accounted for by A after having accounted for everything else, and how much of the residual variability in Y can be accounted for by B after having accounted for everything else as well, and so on. (Note that both go both first and last simultaneously; if this makes sense to you, and accurately reflects your research question, then use type III SS.)

Solved – When parameters are dropped from fixed effects in lmer, drop corresponding random effects

I suggest something like this:

newd = transform(d, ab = interaction(a, b))
m2 = lmer(y ~ ab + (1 + ab | k), data = newd)
lsm = lsmeans(m2, ~ ab)
contrast(lsm, list(c1 = c(...), c2 = c(...), ...))

In other words, use the new factor ab, which consists of the combinations of a and b that actually occur in your dataset, as the one factor to use in the model in place of every usage of a and b. In the contrast call, put the coefficients for meaningful comparisons or contrasts among the levels of ab.