Solved – Internal consistency reliability in item response theory models

item-response-theorypsychometricsreliability

How can internal consistency reliability of a test and of individual test items be quantified in Item Response Theory models? I know I can resort to Classical Test Theory, Cronbach's alpha, and other measures, but is there a way to characterize reliability within IRT?

Best Answer

You can compute test information curves from your IRT parameter estimates. These curves give you the precision of the test at each $\theta$ of the latent trait. The information $I$ can be transformed into the standard error of estimate $SEE$, which is a direct estimate of the reliability of the test at that $\theta$: $SEE = 1 / \sqrt{I}$.

The metric of the test information can also be converted to a traditional reliability metric expressed by a correlation coefficient (Thissen, 2000): $Rel = 1 - (1/I)$. Here are the conversions from a set of TICs to correlational reliability estimates:

# following Thissen, 2000:
TIC <- seq(1, 12, by=1)
round((rel <- data.frame(TIC, SEE=sqrt(1/TIC), REL=1-1/TIC)), 2)

TIC  SEE  REL
  1 1.00 0.00
  2 0.71 0.50
  3 0.58 0.67
  4 0.50 0.75
  5 0.45 0.80
  6 0.41 0.83
  7 0.38 0.86
  8 0.35 0.88
  9 0.33 0.89
 10 0.32 0.90
 11 0.30 0.91
 12 0.29 0.92

For example, a TIC > 5 corresponds to a reliability > .80.

Thissen, D. (2000). Reliability and measurement precision. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2nd ed., pp. 159–184). Lawrence Erlbaum Associates Publishers.

Related Question