I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195).
Comparing the deviance scores (as well as resulting AIC/BIC scores), I know that the 8-dimensional model is superior in a relative sense.
How do I know if this model describes the data well enough in an absolute sense? My idea would be to calculate fit indices like GFI, NNFI etc. but I don't have a clue how to get them from my Conquest output.
Best Answer
Although I am not familiar with Conquest, as it is computing deviance scores and AIC/BIC scores it is likely computing a log-likelihood (LL) that you can use to compute what is usually called a "pseudo r-squared" statistic. For example, McFadden's rho-squared is given by:
rho^2 = 1 - LL_model / LL_null
where LL_model is the log-likelihood of your model and LL_null is the log-likelihood computed for a simple null model (e.g., one which only has intercepts).
If you cannot find the log-likelihood on the output you should be able to compute it backwards from the AIC (i.e., LL - AIC/-2 + k, where k is the number of parameters).
Also, keep in mind that although computing a pseudo r-squared will give you an absolute measure of fit there is, ultimately, no sensible criteria for determining whether it fits the data "well enough". Although there are some problems where a specific level of accuracy may be required, in most applications of statistics (and science) a good model is one that performs better than the other models, and the AIC and BIC are more appropriate for answering such questions than is an absolute measure of fit.