I am doing a research validating a new questionnaire, which has 156 items divided up in 12 scales. I have run a factor analysis at scale level, which gives me two nice constructs (consistent with theory).
My tutor however is insisting this is no good as according to her I need to run an exploratory factor analysis at item level. I have done this and found 43 factors (only a couple of values in this huge table have an absolute square value greater than 0.4, which is the value suggested by Field (2005) as being meaningful.
Is it possible that when running such an analysis with a questionnaire with 100+ items, a factor analysis is really not that appropriate?
I have also been reading the PAI manual – PAI Structure chapter of the PAI questionnaire development (pp. 275-289), as this is a questionnaire that has been developed with lots of funds for research and it is now widely used. No factor analysis has been done on the items, but just at scale level. Several subsequent factor analysis carried out by other authors have also just included scales (not items).
I hope to hear some other thoughts on this, ideally with references to study/theories.
Best Answer
In this answer I will not differentiate between Factor Analysis (FA) and Principal Component Analysis (PCA), but by default I mean PCA. These two are different. In my environment, when someone says "I do factor analysis" they always do mean PCA, almost never realizing the (subtle) difference.
Analysis on both items and scales can be seen as correct and not interchangeable, as they concern a little different problem. People in psychology usually do item-based FA, and probably that's why your tutor (maybe a little too automatically) asks you do for it. Here are the important differences:
My advise:
Do as your tutor asks - FA on items is also a good and valid procedure.
I doubt your 43 factors is a valid result of a factor analysis - to me it more sounds like a upper bound.
A proper FA rarely ends in number of factors equal to an upper bound given by the Kaiser criterion (leave all factors with eigenvalue greater than one). The procedure calls you to consider all possible factor sets (honouring the Kaiser criterion and possibly VariMax-rotated) for which you could give a good name/meaning for each and every factor found. I usually end up in a solution with half the factors than the upper bound. And analysing so many sets factors (where first set contains 43 factors) is a hard work, which hardly can be automatized (except maybe for a deep neural network ;-) ).
What works for me the best is starting from the factor analysis with maximum number of factors and working my way down until I find either a set of factors for which I can give a clear meaning or reach a scree criterion (inflection point of the scree plot) that gives a left bound for number of factors.
Timothy Brown - Confirmatory Factor Analysis for Applied Research, page 23:
If you want to test the theory that your questionnaire has exactly two factors - use a Confirmatory Factor Analysis (it is a special case of path analysis AKA Structural Equation Modeling (SEM) ). But that is a different story.