Solved – the relationship between scale reliability measures (Cronbach’s alpha etc.) and component/factor loadings

cronbachs-alphafactor analysispsychometricsreliabilityscale-construction

Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research.

I know a common approach here is to check the reliability of the scales using Cronbach's alpha or something similar, then aggregate the items in the scales to form scale scores and continue analysis from there.

But there's also factor analysis, which can take all of your item scores as input and tell you which of them form consistent factors. You can get a sense of how strong these factors are by looking at loadings and communalities, and so on. To me this sounds like the same kind of thing, only much more in-depth.

Even if all your scale reliabilities are good, an EFA might correct you on which items fit better into which scales, right? You're probably going to get cross loadings and it might make more sense to use derived factor scores than simple scale sums.

If I want to use these scales for some later analysis (like regression or ANOVA), should I just aggregate the scales so long as their reliability holds up? Or is something like CFA (testing to see if the scales hold up as good factors, which seems to be measuring the same thing as 'reliability').

I've been taught about both approaches independently and so I really don't know how they relate, whether they can be used together or which one makes more sense for which context. Is there a decision tree for good research practice in this case? Something like:

  • Run CFA according to predicted scale items

    • If CFA shows good fit, calculate factor scores and use those for analysis.
    • If CFA shows poor fit, run EFA instead and take exploratory approach (or something).

Are factor analysis and reliability testing indeed separate approaches to the same thing, or am I misunderstanding somewhere?

Best Answer

I am going to add an answer here even though the question was asked a year ago. Most people who are concerned with measurement error will tell you that using factor scores from a CFA is not the best way to move forward. Doing a CFA is good. Estimating factor scores is ok as long as you correct for the amount of measurement error associated with those factor scores in subsequent analyses (a SEM program is the best place to do this).

To get the reliability of the factor score, you need to first calculate the latent construct's reliability from your CFA (or rho):

rho =  Factor score variance/(Factor score variance + Factor score standard
error^2).

Note that the factor score standard error^2 is the error variance of the factor score. This information can be had in MPlus by requesting the PLOT3 output as part of your CFA program.

To calculate overall reliability of the factor score, you use the following formula:

(1-rho)*(FS variance+FS error variance).

The resulting value is the error variance of the factor score. If you were using MPlus for subsequent analyses, you create a latent variable defined by a single item (the factor score) and then specify the factor score's reliability:

LatentF BY FScore@1;
FScore@(calculated reliability value of factor score)

Hope this is helpful! A great resource for this issue are the lecture notes (lecture 11, in particular) from Lesa Hoffman's SEM class at the University of Nebraska, Lincoln. http://www.lesahoffman.com/948/

Related Question