Solved – How to calculate absolute fit indices (RMSEA, GFI…) from relative ones (AIC, BIC…)

aicbicgoodness of fititem-response-theorymodel comparison

I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195).

Comparing the deviance scores (as well as resulting AIC/BIC scores), I know that the 8-dimensional model is superior in a relative sense.

How do I know if this model describes the data well enough in an absolute sense? My idea would be to calculate fit indices like GFI, NNFI etc. but I don't have a clue how to get them from my Conquest output.

Best Answer

Although I am not familiar with Conquest, as it is computing deviance scores and AIC/BIC scores it is likely computing a log-likelihood (LL) that you can use to compute what is usually called a "pseudo r-squared" statistic. For example, McFadden's rho-squared is given by:

rho^2 = 1 - LL_model / LL_null

where LL_model is the log-likelihood of your model and LL_null is the log-likelihood computed for a simple null model (e.g., one which only has intercepts).

If you cannot find the log-likelihood on the output you should be able to compute it backwards from the AIC (i.e., LL - AIC/-2 + k, where k is the number of parameters).

Also, keep in mind that although computing a pseudo r-squared will give you an absolute measure of fit there is, ultimately, no sensible criteria for determining whether it fits the data "well enough". Although there are some problems where a specific level of accuracy may be required, in most applications of statistics (and science) a good model is one that performs better than the other models, and the AIC and BIC are more appropriate for answering such questions than is an absolute measure of fit.

Related Question