I have some ordinal response $y$ that I modeled using both ordinal logistic regression and multinomial logistic regression (to avoid the proportional odds assumption), using two continuous variables as predictors $x_1$ and $x_2$.
I tested different models by including the predictor variables one at a time and their interactions.
$$y \sim x_1 \\
y \sim x_2 \\
y \sim x_1 + x_2 \\
y \sim x_1 + x_2 + x_1x_2
$$
In this way I obtained 8 different models (4 models using ordinal, and 4 models using multinomial logistic regression) and therefore 8 AIC values. It turn out that the best model (the difference in AIC is like 200) is the multinomial logistic with the following predictors:
$$
y \sim x_1 + x_2 + x_1x_2
$$
As far as I know, this model provides the best fit amongst all the various options. Now, how do I quantify if this model is good for the data in an absolute sense (in order for it to be published)?
I can do the regression both using frequentist glm and bayesian glm (so far I did frequentist way because it was more computationally cheap). Ideally I'd like to have the methodology that is most honest and convincing.
EDIT:
I'm more interested to assess model fit in terms of inference rather than prediction. In my specific problem, one of the classes is much more probable than the others on a wide range of the parameters, so prediction is unfeasible. But I'm still interested in discovering how appropriate are the estimates of the underlying probabilities.
Best Answer
Let's take apart your modeling approach to see if we can figure out why a certain model is going to "fit" better.
I would recommend using more comprehensive fit statistics - while the AIC is really good, it is not the only one out there (have you used the BIC as well?). Use Wald tests or LR tests to compare the interaction effects, looking into cross-validation for overall predictive ability, test the parallel lines assumptions. Also think about it theoretically - do you think your measures are ordinal? Does an interaction term make sense? Depending on the field, many journals (particularly in the social sciences) look down on purely data driven modeling approaches if you don't have strong theory to support your decisions.