I have a model M calculated via lme4's glmer function, with random effects ("Customer ID") and fixed effects for each customer ID. My dataset is very large, so I would like to select a sample of Customer IDs, calculate the corresponding fixed and random effect coefficients and then use this model to estimate random effects for the other Customer IDs that haven't been sampled yet.
My main question is, what techniques or packages in R could I use to validate my model? In other words, I want to make sure that my fixed effect estimate is good enough before calculating the random effects of the rest of my dataset.
In general, what techniques exist for splitting datasets and doing gneralized linear models on each subset, finally combining the models?
I would sincerely appreciate references. The best I could find is this.
Best Answer
One of the ways of thinking about random effects (see also this answer) is that they apply to groups that are random draws from population. So if you studied students performance across different schools you could treat schools as either fixed effect with estimating parameter for each school, or as random effect and be interested in overall influence of schools, that can be described by a statistical distribution with its own mean and standard deviation, where individual schools are samples from this distribution. This means that if you are interested in estimating random effects for participants given some random sample from the population, than this seems to agree with the general way of thinking about what random effects describe.
The only issue in here is how well does your sample reflect the population of interest. Now, if you are sampling from a dataset then you have full control on the sampling process. Sampling cases randomly from your population in most cases should be enough for your sample to be representative as long as the sample is big enough (how big is big enough is a different question that you have to ask yourself). You have to remember however that sampling data with hierarchical structure can be more complicated than simple random sampling of cases.
As about validating your model and literature on this topic I would recommend you a book by Gelman and Hill (2006). This book describes linear regression, multilevel models and Bayesian hierarchical models. Authors describe several ways of validating models including approach borrowed from Bayesian statistics named posterior predictive checks (cf. Kruschke, 2013). The idea about posterior predictive checks is simple: you compare posterior distribution of your model with real data to check if it is similar and where they disagree. In non-Bayesian analysis you do not have any posterior distribution, so you can obtain it using a simulation (lme4 library has
simulatefunction for that). The aim of simulation is to produce fake data under fitted model, so this data can be compared to real data. Results of such simulation can be compared visually (e.g. histograms), by using summary statistics (e.g. mean, median, variance, quantiles). Notice also that since you can obtain other samples from your population you can always compare (a) distribution of your sample to the distribution of other samples, (b) posterior distribution of fitted model to distribution of variable of interest in other samples. You should not forget about general model diagnostics, but this is already described in this thread (see also Bates, 2010 and Bolker et al 2008).Combining models (model averaging, cf. Buckland, Burnham and Augustin, 1997) is something that always can be done and is often done. If you are interested in prediction then averaging parameters or predictions from different models should lead to better prediction (in terms of errors) than any of the individual predictions alone and should be more robust than the individual predictions. You can find some brief information about model averaging in papers by Johnson and Omland (2004) and Bolker et al (2008) and more detailed description by Zhang, Zou and Liang (2014), who propose to use information criteria such as AIC for creating weights for averaging (the more information model provides, the greater its weight, similar example here). For averaging, $k$-th models weight $w_k$ is calculates using its AIC value $I_k$ and normalized so weights sum to $1$:
$$ w_k = \frac{\exp(-I_k/2)}{ \sum_{i=1}^K \exp(-I_i/2)} $$
In your case it would be probably enough to take sample from your population, estimate your model on it, and then use this model to make predictions on data from another sample, i.e. use cross-validation with holdout sample. If your model appears to fit badly to the holdout sample data, then you can always make changes in your model (or take larger sample for training model) and assess the results on another holdout sample (better take a different sample than using previous one so not to get overfitting model). Such approach would be easy, not computationally intensive and clear in its methodology. Making predictions on holdout sample would enable you to estimate models error.
Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
Kruschke, J. K. (2013). Posterior predictive checks can and should be Bayesian: Comment on Gelman and Shalizi,‘Philosophy and the practice of Bayesian statistics’. British Journal of Mathematical and Statistical Psychology, 66(1), 45-56.
Bates, D.M. (2010). lme4: Mixed-effects modeling with R. (Unpublished.)
Bolker, B.M., Brooks, M.E., Clark, C.J., Geange, S.W., Poulsen, J.R., Stevens, M.H.H., & White, J.S.S. (2009). Generalized linear mixed models: a practical guide for ecology and evolution. Trends in ecology & evolution, 24(3), 127-135.
Johnson, J. B., & Omland, K. S. (2004). Model selection in ecology and evolution. Trends in ecology & evolution, 19(2), 101-108.
Zhang, X., Zou, G., & Liang, H. (2014). Model averaging and weight choice in linear mixed-effects models. Biometrika, 101(1), 205-218.
Buckland, S.T., Burnham, K.P., & Augustin, N.H. (1997). Model selection: an integral part of inference. Biometrics, 603-618.