I am working on an analysis using exploratory factor analysis (EFA) with the common factor model. This question concerns a methodological issue. Any insights from someone who knows about EFA would be appreciated.
We measured the wellbeing of people using a 15 question survey. The same questionnaire was given the same 100 subjects at 3 different times (there were large gaps in time between when the surveys were administered, therefore we are assuming there was no time dependence [such as carry-over effect]). The goal is to use factor analysis to determine a fixed set of common factors and then calculate the factor scores and to compare the scores of subjects between time points.
Below is my strategy for the analysis. Does this sound reasonable?
Analysis Strategy:
Combine all the data as it if were from a single location. That is to say, create a data set with 100 observations from time 1 and append the 100 observations from time point 2. Then append the 100 observations from time point 3. This would give a combined data set with a total of 300 observations.
Using the above data, fit an EFA model using all 15 variables and 300 observations and obtain factors. After computing the factor scores, again treat the data as if it were from 3 different time points (i.e. 100 subjects measured at three time points). Analyze the factors scores with i.e. ANOVA to test if factor scores of a factor change between the time points.
Best Answer
EFA is not the main issue here, you need to think long and hard about the meaning of correlations between your variables/questions.
Your willingness to assume independence is doing all the work and feels like a way to sidestep a difficult problem through wishful thinking. I don't see how it can be reasonable, even with 6 months in between. That would mean assuming that someone's “well-being” is prone to arbitrary changes, with no link to personality or life circumstances because those are changing much more slowly. You can't expect that type of behavior from psychological variables.
Note that personal attributes are not the only trouble here. Assuming independence would also require that changes between the three time points are essentially random with no relation with any other conceivable variable, thus making comparisons useless.
Imagine that you are drawing a different random sample of 100 participants every six months. If you think the population's distribution on your measure could change in the meantime, ratings collected at a given point in time would be more similar to each other than to ratings collected at other times and these correlations would interact with interpersonal differences in unpredictable ways making correlations computed in the way you describe completely uninterpretable.
This aspect of the assumption is not necessarily completely unreasonable (e.g. intelligence or personality are not supposed to change with the season) but you are probably not willing to make it since you do plan to look at changes between time points.