I've carried out an exploratory factor analysis on a set of survey questions. The result is three factors, however the combination of variables in the third factor makes no sense theoretically. The first two factors do load on these variables (loadings are about .2). I'm going to use the factor scores as variables in further analyses.
Should I drop the variables in the third factor from the analysis, and redo the factor analysis with only two factors? Or keep the variables in the analysis, and just report (and use) two factors?
Best Answer
I would argue that theory and data need to be considered in tandem. Instead of assuming that there are three factors based on theory alone, use the data (e.g., parallel analysis) to suggest what range of factor numbers fits the data. Run and examine EFA models for each number in that range. Use a combination of theory and data-based techniques (e.g., model fit) to decide between the options in that range. If two factor solution seems to make the most sense, then retain it. When you add an additional factor to a model that already accounts for the data pretty well, it is common for a "strange" set of loadings to come out on that additional factor (maybe due to shared methodological variance or noise). If your goal is to understand the latent structure underlying all of your items, then you probably shouldn't trim any out. However, if your goal is to get as clean a model as possible (with all items loading strongly on a single factor) and it isn't a big deal to exclude some, then feel free to trim an item out based on low loadings or high cross-loadings. Make sure to document the whole process as well.
You may find this article helpful. It is older now, but largely still accurate/relevant: Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.