Solved – Model selection: can I compare the AIC from models of count data between linear and poisson models

aicgeneralized linear modelglmmmodel selection

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my case. I selected the model with the best AIC. Finally, I was interested how a linear model (GLM with Gaussian distribution and identity link) would perform in comparisson to the count models. The linear model obviously does not fulfil the distributional assumptions as well as the count models. However, now I find model AICs of arround -150 suggesting much better fit.

Should I use the linear model based on AIC despite the fact that basic distributional assumptions are not met? Are the AICs comparable?

I used a GLMM function for fitting: http://glmmadmb.r-forge.r-project.org/ (my data are clustered).

Best Answer

You cannot use likelihood-based statistics like AIC to compare across models with different likelihood functions - the underlying formulas are different. In linear regression, the likelihood function is the normal density function, in Poisson regression it is the Poisson function. That will account for the differences in the AIC probably more than any differences in fit.

Before you decide to even use a linear model, you need to make sure that the residuals from the model are normally distributed (you can proxy that by looking at the distribution of the outcome variable, though keep in mind it isn't the same). If they are not normally distributed, or close enough for the eye, then you can't use a normal regression model to do any hypothesis testing.

Assuming that it is approximately normal, I would take a two broad approaches to choose the model to report.

1) Predicted outcomes. Estimate the predicted outcomes of each model and compare. Does the linear model have better predictive ability? You may want to do this in a cross-validation framework, where you "train" your model on part of your data and use the rest for prediction.

2) Intuitive interpretation of coefficients. Poisson coefficients can be complicated to understand - they are not the change in number of y but rather a proportional change. Depending on your context this may be more or less useful. Sometimes it is worth sacrificing fit if your model can be more easily interpreted by the end-user - for example, some researchers are willing to avoid the complexity of logit and probit models for the easier-to-interpret coefficients in a linear probability model, even though the LPM has tons of setbacks. Think about who your audience is, what is your context, what is your research question, etc., as you make these decisions.

EDIT: I forgot to add this paper, which gives a good comparison across a range of different count models and may be helpful.