Solved – How to report a linear mixed effects model for those who are unfamiliar and skeptical

lme4-nlmemixed modelreporting

I've run into this issue a few times now, with reviewers requesting more justification for the use of LMMs, traditional tests instead of or in addition to LMMs, and full tables of parameter estimates akin to what you would report with a regular linear model.

Right now my specific issue is a reviewer requesting "A table containing the main parameter estimates of the various models". I am thinking they want something like a traditional table one would report for a linear model (with t tests and p values), but in this case the analyses involve nested model comparisons and there are no t tests for each of the parameters included in each model, but rather a single test for the model comparison, which I do report in the paper. So I'm not sure what to do — I want to satisfy the reviewer, but I don't necessarily want to include huge tables of information that are of little use to evaluating the results. Right now I simply report the beta, SE, chi-square and p value. I also make it clear what variables were included in each model. Any suggestions for how to proceed?

Here is what I am proposing to respond:

We believe the reviewer is asking for something akin to what would be
reported in a traditional multiple regression analysis, with parameter
estimates and their accompanying statistics and p values for each
variable included in a given model. However, because linear mixed
model analyses use nested models comparing reduced models to full
models with one additional parameter, the only parameter that is
tested is the one that is added in the full model (CITATION) As such,
including a table would not support interpretation of the results in
the way that it would in a more traditional analysis. Thus for each
analysis, we report the betas for the tested parameter in each model
comparison, along with the key statistics, in the body of the results
section, as is recommended (CITATIONS).

Also, when asked for a justification for the use of LMMs in my particular case, this is what I'm proposing to respond:

We used linear mixed models because this analysis allowed us to
account for variability due to trial type in our models (switch versus
no-switch trials), while simultaneously accounting for the fact that
trials were nested within subjects, and multiple responses from the
same person are more similar than responses from other people.
Accounting for both trial type and subject-level variance in reaction times was expected to reduce error in our models and increase our ability
to detect any effect of task performance.

If you have any suggestions for how this could be improved, I'd appreciate it. Again, this audience is not statistically sophisticated, so adding tables and supplementary data is only likely to add to their confusion/skepticism.

Also, note that my motivation for using LMMs is different from what I've seen in papers (e.g., modeling multiple random effects simultaneously – in my case, there's only one random effect – participants, and trial type is a fixed effect), so I am not sure that citing some of the common papers is that helpful. It's possible that I've overlooked other ways to analyze this data so my justification for using LMMs is not apt.

Best Answer

I partly take side with the reviewer on this one. You are interested in the effect of your parameter of interest — given the rest of the model. It is hard to interpret the results and to check the the validity of the model if you only report a single parameter of interest. I would provide:

the formula of your model
beta estimates for all fixed effects
corresponding SEs and CIs
corresponding test statistics (z, t, Chi^2, change in AIC/BIC, whatever you used) with df's/n's
corresponding p values
SDs for your random effects and their correlations (if necessary as separate table)

The space constraints in most classical journals will make it necessary to put these information into an online supplement.

Examples for reporting mixed models can be found here.

Related Solutions

Solved – Reporting results of linear mixed-effects model

This may not help answer your question, but I noticed that you have a repeated measure (Day) in your experiment, but you did not indicate that this was a repeated measure in your model. I would have thought the random term in your model to be as such:

mymodel <- lme(dv ~ Treatment*Day, random = ~1|Subject/Day, 
               data = mydf, na.action = na.omit,
               correlation = corAR1(form = ~1 |Subject/Day), method = "REML")

As for reporting the results, did you intend to report on the day on which you start seeing significant differences between the treatments? If so, then I think you'll need to look at/report on the contrasts on the interaction term as well. I'm a stats novice myself and basically have the same question as you do :-)

Andy Field's "Discovering Statistics Using R" explains how to report results from a linear mixed effects model in Ch14. I don't have the book at hand but can edit this post once I get my hands on it again.

Solved – How to choose random- and fixed-effects structure in linear mixed models

I'm not sure there's really a canonical answer to this, but I'll give it a shot.

What is the recommended way to select the best fitting model in this context? When using log-likelihood ratio tests what is the recommended procedure? Generating models upwards (from null model to most complex model) or downwards (from most complex model to null model)? Stepwise inclusion or exclusion? Or is it recommended to put all models in one log-likelihood ratio test and select the model with the lowest p-value? How to compare models that are not nested?

It depends what your goals are.

In general you should be very, very careful about model selection (see e.g. this answer, or this post, or just Google "Harrell stepwise" ...).
If you are interested in having your p-values be meaningful (i.e. you are doing confirmatory hypothesis testing), you should not do model selection. However: it's not so clear to me whether model selection procedures are quite as bad if you are doing model selection on non-focal parts of the model, e.g. doing model selection on the random effects if your primary interest is inference on the fixed effects.
There's no such thing as "putting all the models in one likelihood ratio test" -- likelihood ratio testing is a pairwise procedure. If you wanted to do model selection (e.g.) on the random effects, I would probably recommend an "all at once" approach using information criteria as in this example -- that at least avoids some of the problems of stepwise approaches (but not of model selection more generally).
Barr et al. 2013 "Keep it maximal" Journal of Memory and Language (doi:10.1016/j.jml.2012.11.001) would recommend using the maximal model (only).
Shravan Vasishth disagrees, arguing that such models are going to be underpowered and hence problematic unless the data set is very large (and the signal-to-noise ratio is high)
Another reasonably defensible approach is to fit a large but reasonable model and then, if the fit is singular, remove terms until it isn't any more
With some caveats (enumerated in the GLMM FAQ), you can use information criteria to compare non-nested models with differing random effects (although Brian Ripley disagrees: see bottom of p. 6 here)

Is it recommended to first find the appropriate fixed-effects structure and then the appropriate random-effects structure or the other way round (I have found references for both options...)?

I don't think anyone knows. See previous answer about model selection more generally. If you could define your goals sufficiently clearly (which few people do), the question might be answerable. If you have references for both options, it would be useful to edit your question to include them ... (For what it's worth, this example (already quoted above) uses information criteria to select the random effects part, then eschews selection on the fixed-effect part of the model.

What is the recommended way of reporting results? Reporting the p-value from the log-likelihood ratio test comparing the full mixed-model (with the effect in question) to reduced model (without the effect in question). Or is it better to use log-likelihood ratio test to find the best fitting model and then use lmerTest to report p-values from the effects in the best fitting model?

This is (alas) another difficult question. If you report the marginal effects as reported by lmerTest, you have to worry about marginality (e.g., whether the estimates of the main effects of A and B are meaningful when there is an A-by-B interaction in the model); this is a huge can of worms, but is somewhat mitigated if you use contrasts="sum" as recommend by afex::mixed(). Balanced designs help a little bit too. If you really want to paper over all these cracks, I think I would recommend afex::mixed, which gives you output similar to lmerTest, but tries to deal with these issues.

Best Answer

Related Solutions

Solved – Reporting results of linear mixed-effects model

Solved – How to choose random- and fixed-effects structure in linear mixed models

Related Question