In addition to proc varclus, randomForest, and assessing multicollinearity among potential predictor variables, I am seeking other methods of variable selection in lieu of using stepwise methods for building more parsimonious binary logistic regression models from a wide array of potential predictor variables. I have done some research into other methods such as Mutual Information (MI), and I have two questions in regards to its use:
1) Has anyone used MI for binary logistic regression variable selection? If so, what are your thoughts on its application?
2) Does anyone know how to calculate MI using either Base SAS or R for potential predictor variables in reference to the outcome of interest? Any help or references in this area would be greatly appreciated!
Thanks!
Best Answer
As I expected, no one probably uses MI to perform variable selection, and thus I will stick with other methods such as variable clustering, random forest, PCA, correlation assessment among potential predictor variables, and other methods in lieu of using MI to deal with high dimensionality.
Additionally, MI is likely not that useful for continuous variables, which encompass a large percentage of the candidate predictor variables in a typical dataset that I work with, so I am likely much better off without making using of MI as another variable selection technique even if it is a worthwhile approach to do so with continuous potential predictor variables.
PLEASE correct me if I am wrong, as I would enjoy exploring a new technique (at least to me) of avoiding the use of stepwise techniques to reduce the number of candidate variables in my logistic regression models!