Solved – Comparing importance of different sets of predictors

importancepredictorregression

I was advising a research student with a particular problem, and I was keen to get the input of others on this site.

Context:

The researcher had three types of predictor variables. Each type contained a different number of predictor variables. Each predictor was a continuous variable:

  • Social: S1, S2, S3, S4 (i.e., four predictors)
  • Cognitive: C1, C2 (i.e., two predictors)
  • Behavioural: B1, B2, B3 (i.e., three predictors)

The outcome variable was also continuous.
The sample included around 60 participants.

The researcher wanted to comment about which type of predictors were more important in explaining the outcome variable. This was related to broader theoretical concerns about the relative importance of these types of predictors.

Questions

  • What is a good way to assess the relative importance of one set of predictors relative to another set?
  • What is a good strategy for dealing with the fact that there are different numbers of predictors in each set?
  • What caveats in interpretation might you suggest?

Any references to examples or discussion of techniques would also be most welcome.

Best Answer

Suggestions

  • You could perform individual multiple regressions for each type of predictor, and compare across multiple regressions, adjusted r-square, generalised r-square, or some other parsimony adjusted measure of variance explained.
  • You could alternatively explore the general literature on variable importance (see here for a discussion with links). This would encourage a focus on the importance of individual predictors.
  • In some situations hierarchical regression may provide a useful framework. You would enter one type of variable in one block (e.g., cognitive variables), and in the second block another type (e.g., social variables). This would help answer the question of whether one type of variable predicts over and above another type.
  • As a side examination, you could run a factor analysis on the predictor variables to examine whether the correlations between predictor variables map on to the assignment of variables to types.

Caveats

  • Types of variables such as cognitive, social, and behavioural are broad classes of variables. A given study will always include only a subset of the possible variables, and typically such a subset is small relative to the possible variables. Furthermore, the measured variables may not be the most reliable or valid means of measuring the intended construct. Thus, you need to be careful when drawing the broader inference about the relative importance of a given type of variable over and beyond what was actually measured.
  • You also need to consider any bias in the way that the dependent variable was measured. Particularly in psychological studies, there is a tendency for self-report measures to correlate well with self-report, ability with ability, other-report with other report, and so on. The issue is that the mode of measurement has a large effect over and beyond the actual construct of interest. Thus, if the dependent variable is measured in a particular way (e.g., self-report), then don't over-interpret larger correlations with one type of predictor if that type also uses self-report.
Related Question