Solved – How to test whether ICCs from different groups are significantly different

intraclass-correlation

I need to compare two generalizability (G) coefficients for data that are from two separate populations. G coefficients are a type of intraclass correlation coefficient (ICC).

The literature on statistically comparing G coefficients is sparse by all accounts, so I thought that the best I might do was find out the recommendation for statistically comparing ICCs. I've considered Fisher's Z, but I know ICCs are not normally distributed.

Best Answer

Assuming that the generalizability coefficients that you obtained involve overlapping measurement methods in the two separate populations (in other words there is non-independence between the two coefficients) you should be able to compare them directly in PROC GLM or MIXED as detailed in this Book Chapter by Cohen and Doveh (2005).

Reference

Cohen, A., & Doveh, E. (2005). Significance tests for differences between dependent intraclass correlation coefficients (ICCs). Research in Multi Level Issues, 4, 373-420.

Related Solutions

Solved – Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties

These are distinct ways of accounting for raters or items variance in overall variance, following Shrout and Fleiss (1979) (cases 1 to 3 in Table 1):

One-way random effects model: raters are considered as sampled from a larger pool of potential raters, hence they are treated as random effects; the ICC is then interpreted as the % of total variance accounted for by subjects/items variance. This is called the consistency ICC.
Two-way random effects model: both factors -- raters and items/subjects -- are viewed as random effects, and we have two variance components (or mean squares) in addition to the residual variance; we further assume that raters assess all items/subjects; the ICC gives in this case the % of variance attributable to raters + items/subjects.
Two-way mixed model: contrary to the one-way approach, here raters are considered as fixed effects (no generalization beyond the sample at hand) but items/subjects are treated as random effects; the unit of analysis may be the individual or the average ratings.

I would say raters have to be entered as columns, although I'm not a specialist of SPSS. Dave Garson's dedicated website is worth looking at for those working with SPSS. There is also a complete on-line tutorial on reliability analysis (Robert A. Yaffee) [archived version].

For theoretical consideration about the mixed-effect approach, please consider reading my answer to this related question: Reliability in Elicitation Exercise.

Solved – Partitioning variance within a level in a 3-level mixed effects model

I figured out a workaround, although the residual variance is not parsable in lme4 (as far as I know). One workaround is to create separate numeric dummy variables for white and rem. I made these white_dummy = 1 and another column rem_dummy = 1. Then I ran the following model:

model3 <- lmer(outcome ~ rem + 
 (0 + white_dummy|id) +
 (0 + rem_dummy|id) +
 (0 + white_dummy|provider) +
 (0 + rem_dummy|provider), 
 data = filter(dat, rem == 1 | rem == 0))

and this creates the following random effects output:

Random effects:
 Groups        Name         Variance Std.Dev. Corr
 id            white_dummy0 0.19631  0.4431       
               white_dummy1 0.39760  0.6306   0.06
 id.1          rem_dummy0   0.18381  0.4287       
               rem_dummy1   0.39400  0.6277   0.24
 provider_1    white_dummy0 0.06192  0.2488       
               white_dummy1 0.01334  0.1155   0.11
 provider_1.1  rem_dummy0   0.02774  0.1666       
               rem_dummy1   0.03867  0.1966   0.00
Residual                    0.17943  0.4236       
Number of obs: 8606, groups:  id, 1504; therapist_1, 39

However! The model failed to converge and this is likely due to the reduced number of observations at the therapist/rem level and the variance estimates are near zero. I am going to go the bayesian MCMCglmm route to attempt to fit this model.

Best Answer

Reference

Related Solutions

Solved – Inter-rater reliability using Intra-class correlation with ratings for multiple objects on multiple properties

Solved – Partitioning variance within a level in a 3-level mixed effects model

Related Question