Let us look at some sample data for 5 hypothetical subjects.
library(emmeans)
library(lme4)
# generate some sample data
# condition (Placebo, Treatment)
# type (some factor, e.g. two different medications)
# value (outcome of interest)
# subject
data <- data.frame(condition = as.factor(rep(c(0,1), times = 10)), value = rnorm(20), type = rep(c("A","B"), each = 10), subject = as.factor(rep(rep(1:5, each = 2), times = 2)))
Now we like to conduct a mixed model and post-hoc tests with emmeans
, where we test conditions per type in a pairwise fashion:
# caluculate a mixed model
fit_data <- lmer(value ~ condition*type + (1|subject), data = data)
# emmeans post-hoc test
emmeans(fit_data, pairwise ~ condition | type)
So for each type (A or B), I have paired data (condition 0 has 5 data points and condition 1 has 5 data points). If I would do a paired t-test like so:
# paired t-test
t.test(data[data$condition==0 & data$type=="A","value"], data[data$condition==1 & data$type=="A","value"], paired = TRUE)
t.test(data[data$condition==0 & data$type=="B","value"], data[data$condition==1 & data$type=="B","value"], paired = TRUE)
I would get degrees of freedom of 4 for the paired t-test, but emmeans says the degrees of freedom are 12. Why is there this huge difference? If the emmeans
package also would use df = 4, then the p-values would also be more comparable.
Best Answer
Your
t.test()
results are each based on only selected portions of the data, whereas theemmeans()
results are based on a model that is fitted to all of the data. Consequently, the denominators for the $t$ statistics are based on pooled information in the model. More information, more df.More details
Let me create your data in a reproducible way, by providing the random-number seed:
Here are your t-test results:
Now we compare with
emmeans()
results. The important thing to know aboutemmeans()
is that it provides an interpretation of a fitted model, not of the dataset itself. If you give it a different fitted model, you will get different results. Here is what we get with your model:Each t test shown indeed has 12 d.f., and that is because of information extracted from the model provided it,
mod1
. Now consider a different model: one that is fitted only to the data for type A:This model does not include
type
as a predictor, so of course we have to alter theemmeans()
call accordingly:With model
mod2
, we get only 4 d.f. And, in fact, the t test and P value are identical to what was obtained above usingt.test()
.Again, the point is that
emmeans()
summarizes a fitted model. The modelmod1
uses data from both types, and it incorporates an assumption that the error variance is the same with each type, and that the subject variance is the same with each type. It pools the information from the two types to obtain more accurate* estimates of these variances, and that is reflected in the increased degrees of freedom. [* more accurate, assuming the model assumptions are correct!]Because of this, if you are satisfied, based on diagnostic plots, etc., that
mod1
fits the data adequately, then theemmeans()
results frommod1
are better than those frommod2
because more efficient use is made of available information. If, on the other hand,mod1
does not fit well,mod2
may be better and hence you should give that model toemmeans()
.