How are (linear) mixed effects models normally compared against each other? I know likelihood ratio tests can be used, but this doesn't work if one model is not a 'subset' of the other correct?
Is the estimation of the models df always straightforward? Number of fixed effects + number of variance components estimated? Do we ignore the random effects estimates?
What about validation? My first thought is cross validation, but random folds might not work given the structure of the data. Is a methodology of 'leave one subject/cluster out' appropriate? What about leave one observation out?
Mallows Cp can be interpreted as an estimate of the models prediction error. Model selection via AIC attempts to minimize the prediction error (So Cp and AIC should pick the same model if the errors are Gaussian I believe). Does this mean AIC or Cp can be used to pick an 'optimal' linear mixed effects model from a collection of some un-nested models in terms of prediction error? (provided they are fit on the same data) Is BIC still more likely to pick the 'true' model amongst candidates?
I am also under the impression that when comparing mixed effects models via AIC or BIC we only count the fixed effects as 'parameters' in the calculation, not the actual models df.
Is there any good literature on these topics? Is it worth investigating cAIC or mAIC? Do they have specific application outside of AIC?
Best Answer
The main problem on model selection in mixed models is to define the degrees of freedom (df) of a model, truly. To compute df of a mixed model, one has to define the number of estimated parameters including fixed and random effects. And this is not straightforward. This paper by Jiming Jiang and others (2008) entitled "Fence methods for mixed model selection" could be applied in such situations. A new related work is this one by Greven, S. & Kneib, T. (2010) entitled "On the behavior of marginal and conditional AIC in linear mixed models". Hope this could be helpful.