While helping someone else with their analyses, I've run into a question regarding the difference between t-tests and F-tests for linear mixed models in lme4 for R, as provided by lmerTest. I'm aware of the problems with calculating any kind of p-values for linear mixed models (as I understand, primarily due to the fact that definition of the degrees of freedom is problematic), as well as the problems with interpreting main effects in the presence of significant interactions (based on the marginality principle).
Briefly, the data are from an experiment with two conditions (congruity TRUE/FALSE), measured on six sets of sensors which can be described as a combination of two factors: anteriority (anterior/posterior) and laterality (left/central/right).
As can be seen from the summary output below, the t.tests do not show a significant congruity effect (p = 0.12), while the anova output shows a very significant congruity effect (p = 2.8e-10). Since congruity has only two levels, this cannot be the result of the F-test doing an omnibus test over several levels of the fixed factor. I am therefore unsure what causes the very significant result in the anova output. Is this due to the fact that there are strong interactions involving congruity which of course depend on the inclusion of the main effect in the model parametrization?
I have looked for a previous answer to this question on CrossValidated but I have not been able to find anything relevant except possibly the first answer to this question. However, if that does provide a real answer then it is implicit in the mathematics, and I am looking for a conceptual answer that I can explain to the person I am trying to help.
> final.mod<-lmer(uV~1+factor(congruity)*factor(laterality)*factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(final.mod)
Linear mixed model fit by REML
t-tests use Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) * factor(laterality) * factor(anteriority) + (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 348903.5
Scaled residuals:
Min 1Q Median 3Q Max
-7.0440 -0.6002 0.0069 0.6038 11.3912
Random effects:
Groups Name Variance Std.Dev.
sent.id (Intercept) 1.773 1.332
Subject (Intercept) 2.548 1.596
Residual 111.396 10.554
Number of obs: 46176, groups: sent.id, 41; Subject, 30
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 4.768e-03 3.973e-01 7.900e+01 0.012 0.9905
factor(congruity)TRUE 3.758e-01 2.410e-01 4.611e+04 1.559 0.1189
factor(laterality)left 7.154e-02 2.430e-01 4.610e+04 0.294 0.7685
factor(laterality)right -2.003e-01 2.430e-01 4.610e+04 -0.824 0.4098
factor(anteriority)posterior -4.203e-02 2.430e-01 4.610e+04 -0.173 0.8627
factor(congruity)TRUE:factor(laterality)left -1.013e-01 3.404e-01 4.610e+04 -0.298 0.7660
factor(congruity)TRUE:factor(laterality)right 7.233e-02 3.404e-01 4.610e+04 0.213 0.8317
factor(congruity)TRUE:factor(anteriority)posterior 6.162e-01 3.404e-01 4.610e+04 1.810 0.0702 .
factor(laterality)left:factor(anteriority)posterior 2.568e-01 3.437e-01 4.610e+04 0.747 0.4549
factor(laterality)right:factor(anteriority)posterior 1.763e-01 3.437e-01 4.610e+04 0.513 0.6080
factor(congruity)TRUE:factor(laterality)left:factor(anteriority)posterior -5.162e-02 4.813e-01 4.610e+04 -0.107 0.9146
factor(congruity)TRUE:factor(laterality)right:factor(anteriority)posterior -2.420e-01 4.813e-01 4.610e+04 -0.503 0.6152
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr) fc()TRUE fctr(ltrlty)l fctr(ltrlty)r fctr(n) fctr(cngrty)TRUE:fctr(ltrlty)l fctr(cngrty)TRUE:fctr(ltrlty)r
fctr(c)TRUE -0.310
fctr(ltrlty)l -0.306 0.504
fctr(ltrlty)r -0.306 0.504 0.500
fctr(ntrrt) -0.306 0.504 0.500 0.500
fctr(cngrty)TRUE:fctr(ltrlty)l 0.218 -0.706 -0.714 -0.357 -0.357
fctr(cngrty)TRUE:fctr(ltrlty)r 0.218 -0.706 -0.357 -0.714 -0.357 0.500
fctr(cngrty)TRUE:fctr(n) 0.218 -0.706 -0.357 -0.357 -0.714 0.500 0.500
fctr(ltrlty)l:() 0.216 -0.357 -0.707 -0.354 -0.707 0.505 0.252
fctr(ltrlty)r:() 0.216 -0.357 -0.354 -0.707 -0.707 0.252 0.505
fctr(cngrty)TRUE:fctr(ltrlty)l:() -0.154 0.499 0.505 0.252 0.505 -0.707 -0.354
fctr(cngrty)TRUE:fctr(ltrlty)r:() -0.154 0.499 0.252 0.505 0.505 -0.354 -0.707
fctr(cngrty)TRUE:fctr(n) fctr(ltrlty)l:() fctr(ltrlty)r:() fctr(cngrty)TRUE:fctr(ltrlty)l:()
fctr(c)TRUE
fctr(ltrlty)l
fctr(ltrlty)r
fctr(ntrrt)
fctr(cngrty)TRUE:fctr(ltrlty)l
fctr(cngrty)TRUE:fctr(ltrlty)r
fctr(cngrty)TRUE:fctr(n)
fctr(ltrlty)l:() 0.505
fctr(ltrlty)r:() 0.505 0.500
fctr(cngrty)TRUE:fctr(ltrlty)l:() -0.707 -0.714 -0.357
fctr(cngrty)TRUE:fctr(ltrlty)r:() -0.707 -0.357 -0.714 0.500
> anova(final.mod)
Analysis of Variance Table of type III with Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
factor(congruity) 4439.1 4439.1 1 46142 39.850 2.768e-10 ***
factor(laterality) 572.9 286.5 2 46095 2.572 0.076430 .
factor(anteriority) 1508.1 1508.1 1 46095 13.538 0.000234 ***
factor(congruity):factor(laterality) 31.6 15.8 2 46095 0.142 0.867581
factor(congruity):factor(anteriority) 775.1 775.1 1 46095 6.958 0.008349 **
factor(laterality):factor(anteriority) 111.9 56.0 2 46095 0.502 0.605126
factor(congruity):factor(laterality):factor(anteriority) 31.2 15.6 2 46095 0.140 0.869183
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
In reply to @Aurelie's question:
> congruity.mod<-lmer(uV~1+factor(congruity)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(congruity.mod)
Linear mixed model fit by REML
t-tests use Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) + (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 494077.2
Scaled residuals:
Min 1Q Median 3Q Max
-10.1673 -0.5790 -0.0097 0.5818 12.6088
Random effects:
Groups Name Variance Std.Dev.
sent.id (Intercept) 4.568 2.137
Subject (Intercept) 6.132 2.476
Residual 178.137 13.347
Number of obs: 61568, groups: sent.id, 41; Subject, 30
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.6055 0.5671 57.0000 1.068 0.29
factor(congruity)FALSE -0.7105 0.1084 61535.0000 -6.558 5.51e-11 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr)
fctr()FALSE -0.093
> anova(congruity.mod)
Analysis of Variance Table of type III with Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
factor(congruity) 7660.5 7660.5 1 61535 43.004 5.507e-11 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> laterality.mod<-lmer(uV~1+factor(laterality)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(laterality.mod)
Linear mixed model fit by REML
t-tests use Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(laterality) + (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 372848.2
Scaled residuals:
Min 1Q Median 3Q Max
-9.7033 -0.5981 -0.0076 0.6006 12.2265
Random effects:
Groups Name Variance Std.Dev.
sent.id (Intercept) 5.568 2.360
Subject (Intercept) 6.777 2.603
Residual 186.966 13.674
Number of obs: 46176, groups: sent.id, 41; Subject, 30
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.8128 0.6115 61.0000 1.329 0.18877
factor(laterality)left -0.4260 0.1559 46105.0000 -2.733 0.00628 **
factor(laterality)right -0.6709 0.1559 46105.0000 -4.304 1.68e-05 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr) fctr(ltrlty)l
fctr(ltrlty)l -0.127
fctr(ltrlty)r -0.127 0.500
> anova(laterality.mod)
Analysis of Variance Table of type III with Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
factor(laterality) 3548.2 1774.1 2 46105 9.4889 7.584e-05 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> anteriority.mod<-lmer(uV~1+factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(anteriority.mod)
Linear mixed model fit by REML
t-tests use Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(anteriority) + (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 372738.6
Scaled residuals:
Min 1Q Median 3Q Max
-9.6668 -0.5986 -0.0032 0.6017 12.2711
Random effects:
Groups Name Variance Std.Dev.
sent.id (Intercept) 5.569 2.360
Subject (Intercept) 6.777 2.603
Residual 186.525 13.657
Number of obs: 46176, groups: sent.id, 41; Subject, 30
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) -0.2693 0.6081 59.0000 -0.443 0.66
factor(anteriority)posterior 1.4328 0.1271 46105.0000 11.272 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr)
fctr(ntrrt) -0.105
> anova(anteriority.mod)
Analysis of Variance Table of type III with Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
factor(anteriority) 23700 23700 1 46106 127.06 < 2.2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Update: After updating the contrasts based on @Henrik's answer:
> options(contrasts=c("contr.sum","contr.poly"))
> final.mod<-lmer(uV~1+factor(congruity)*factor(laterality)*factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(final.mod)
Linear mixed model fit by REML
t-tests use Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) * factor(laterality) * factor(anteriority) + (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 372689.8
Scaled residuals:
Min 1Q Median 3Q Max
-9.6772 -0.5979 -0.0016 0.5977 12.3439
Random effects:
Groups Name Variance Std.Dev.
sent.id (Intercept) 5.556 2.357
Subject (Intercept) 6.752 2.599
Residual 186.232 13.647
Number of obs: 46176, groups: sent.id, 41; Subject, 30
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 4.355e-01 6.039e-01 5.800e+01 0.721 0.4737
factor(congruity)1 4.501e-01 6.396e-02 4.613e+04 7.037 1.99e-12 ***
factor(laterality)1 3.628e-01 8.983e-02 4.610e+04 4.039 5.38e-05 ***
factor(laterality)2 -5.732e-02 8.983e-02 4.610e+04 -0.638 0.5234
factor(anteriority)1 -7.183e-01 6.352e-02 4.610e+04 -11.308 < 2e-16 ***
factor(congruity)1:factor(laterality)1 1.433e-01 8.983e-02 4.610e+04 1.596 0.1106
factor(congruity)1:factor(laterality)2 -1.535e-01 8.983e-02 4.610e+04 -1.709 0.0875 .
factor(congruity)1:factor(anteriority)1 9.442e-02 6.352e-02 4.610e+04 1.487 0.1371
factor(laterality)1:factor(anteriority)1 2.282e-01 8.983e-02 4.610e+04 2.540 0.0111 *
factor(laterality)2:factor(anteriority)1 -2.121e-01 8.983e-02 4.610e+04 -2.362 0.0182 *
factor(congruity)1:factor(laterality)1:factor(anteriority)1 -7.802e-03 8.983e-02 4.610e+04 -0.087 0.9308
factor(congruity)1:factor(laterality)2:factor(anteriority)1 -1.141e-02 8.983e-02 4.610e+04 -0.127 0.8989
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr) fctr(c)1 fctr(l)1 fct()2 fctr(n)1 fctr(cngrty)1:fctr(l)1 fc()1:()2 fctr(cngrty)1:fctr(n)1
fctr(cngr)1 -0.003
fctr(ltrl)1 0.000 0.000
fctr(ltrl)2 0.000 0.000 -0.500
fctr(ntrr)1 0.000 0.000 0.000 0.000
fctr(cngrty)1:fctr(l)1 0.000 0.000 -0.020 0.010 0.000
fctr()1:()2 0.000 0.000 0.010 -0.020 0.000 -0.500
fctr(cngrty)1:fctr(n)1 0.000 0.000 0.000 0.000 -0.020 0.000 0.000
fctr(l)1:()1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
fctr()2:()1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
f()1:()1:() 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
f()1:()2:() 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
fctr(l)1:()1 f()2:( f()1:()1:
fctr(cngr)1
fctr(ltrl)1
fctr(ltrl)2
fctr(ntrr)1
fctr(cngrty)1:fctr(l)1
fctr()1:()2
fctr(cngrty)1:fctr(n)1
fctr(l)1:()1
fctr()2:()1 -0.500
f()1:()1:() -0.020 0.010
f()1:()2:() 0.010 -0.020 -0.500
> anova(final.mod)
Analysis of Variance Table of type III with Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
factor(congruity) 9221.9 9221.9 1 46129 49.518 1.993e-12 ***
factor(laterality) 3511.5 1755.7 2 46095 9.428 8.062e-05 ***
factor(anteriority) 23814.0 23814.0 1 46095 127.873 < 2.2e-16 ***
factor(congruity):factor(laterality) 680.3 340.1 2 46095 1.826 0.16101
factor(congruity):factor(anteriority) 411.5 411.5 1 46095 2.210 0.13714
factor(laterality):factor(anteriority) 1497.4 748.7 2 46095 4.020 0.01796 *
factor(congruity):factor(laterality):factor(anteriority) 8.6 4.3 2 46095 0.023 0.97713
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Best Answer
Type III tests require correct coding for lower-order effects to be meaningful, specifically orthogonal contrasts. The R default
contr.treatmentis not orthogonal, other contrasts are though (e.g.,contr.sum). In your code it looks like you used did not change the defaults, hence your results are so-called simple main effects. We discuss this in our soon-to-come out chapter here, but other references are easy to find.To use the correct contrasts run the following before fitting a mixed model in R:
An easier to remember code is to use
set_sum_contrasts()from myafexpackage:Please update your question if this does not resolve your problem (preferably with data to recreate the problem).