Solved – Validation of a scale for a different population (CFA)

confirmatory-factorreliabilityscalesvalidation

I'm currently adapting a coping questionnaire which has been widely used in sports to be used with a different population (musicians), so I need to address its reliability and validity with this new population. I'm a novice in statistics therefore any advice will be very helpful.

The questionnaire has 39 items on a 5 point Likert scale (1=not at all; 5= very much). In sports population, EFA and CFA have shown a 10 first order factors and 3 second order factors. Since I'm unsure if in the new population (musicians) those 10 factors will be the same, I'm wondering how I should proceed to test validity and reliability. My questions are:

1.Should I calculate the Cronbach's $\alpha$ for every 10 first order factor from the original questionnaire in order to test internal consistency (and perhaps delete some items) and then proceed with a CFA? Or should I run an EFA with all 39 items?

Once those X first order factors have been established, will I just need to run a CFA to confirm (or not) that they load in a 3 (perhaps just 2) second order factors?
I'm collecting my responses to the questionnaire right at this time, and I'm expecting to get between 100-120 observations. Is this sample enough to perform those tests or would I need a larger one?

Thank you very much for your advice.

Best Answer

A confirmatory factor analysis would be used to test whether the questionnaire had the same structure among musicians as among athletes. EFA would be used to see what the factor structure is among musicians. Either might be right, but I think the latter is more likely what you want.

But before doing any FA, I'd look at item statistics. Perhaps some items that worked well with athletes do not work with musicians. Are there any items that get very skewed responses? They may not be contributing much. I would also look at all the correlations (39x38/2 of them). I might even make a huge scatterplot matrix to look at all the scatterplots.

Then I'd do EFA.

Related Solutions

Solved – How to reduce number of items using factor analysis, internal consistency, and item response theory in conjunction

I don't have any citations, but here's what I'd suggest:

Zeroth: If at all possible, split the data into a training and test set.

First do EFA. Look at various solutions to see which ones make sense, based on your knowledge of the questions. You'd have to do this before Cronbach's alpha, or you won't know which items go into which factor. (Running alpha on ALL the items is probably not a good idea).

Next, run alpha and delete items that have much poorer correlations than the others in each factor. I wouldn't set an arbitrary cutoff, I'd look for ones that were much lower than the others. See if deleting those makes sense.

Finally, choose items with a variety of "difficulty" levels from IRT.

Then, if possible, redo this on the test set, but without doing any exploring. That is, see how well the result found on the training set works on the test set.

Solved – the relationship between scale reliability measures (Cronbach’s alpha etc.) and component/factor loadings

I am going to add an answer here even though the question was asked a year ago. Most people who are concerned with measurement error will tell you that using factor scores from a CFA is not the best way to move forward. Doing a CFA is good. Estimating factor scores is ok as long as you correct for the amount of measurement error associated with those factor scores in subsequent analyses (a SEM program is the best place to do this).

To get the reliability of the factor score, you need to first calculate the latent construct's reliability from your CFA (or rho):

rho =  Factor score variance/(Factor score variance + Factor score standard
error^2).

Note that the factor score standard error^2 is the error variance of the factor score. This information can be had in MPlus by requesting the PLOT3 output as part of your CFA program.

To calculate overall reliability of the factor score, you use the following formula:

(1-rho)*(FS variance+FS error variance).

The resulting value is the error variance of the factor score. If you were using MPlus for subsequent analyses, you create a latent variable defined by a single item (the factor score) and then specify the factor score's reliability:

LatentF BY FScore@1;
FScore@(calculated reliability value of factor score)

Hope this is helpful! A great resource for this issue are the lecture notes (lecture 11, in particular) from Lesa Hoffman's SEM class at the University of Nebraska, Lincoln. http://www.lesahoffman.com/948/

Best Answer

Related Solutions

Solved – How to reduce number of items using factor analysis, internal consistency, and item response theory in conjunction

Solved – the relationship between scale reliability measures (Cronbach’s alpha etc.) and component/factor loadings

Related Question