Solved – Can the AICc (corrected Akaike information criterion) measure be used for post-hoc analysis

aicmodel selectionpost-hoc

I am collecting a lot of data from neurons and will be analyzing each neuron separately. Is it valid to pick the best of several models based on the AICc for each neuron and then use the ANOVA p-values from the best model? Or should should I only run the AICc on preliminary data and then use the best model analyze every neuron?

My impression was that it is only statistically valid to select the model in the preliminary experimental design phase, but I'm now having difficulty finding this explicitly stated. References would be greatly appreciated.

Best Answer

I believe I have found my own answer. It sounds like using AIC to select from a large set of conceivable variables can result in models that are "excessively" tailored to the data at hand, ie. data dredging. However, if selecting from a limited set of a priori models, then it should be fine.

Reference:

Question 4 from Joseph E. Cavanaugh's slides at: http://myweb.uiowa.edu/cavaaugh/ms_lec_14_ho.pdf