Solved – Generalising correlation-based feature selection

classificationdata miningfeature selectionpredictive-modelsregression

One method to perform feature selection consists in calculating Pearson's correlation coefficient between each explanatory variable $X_i$ and the response variable $Y$. Then the absolute values of the coefficients are sorted in decreasing order and the first $K$ variables are selected.
As far as I know, Pearson's correlation coefficient can be calculated only between two quantitative variables. So:

How can I perform feature selection if the dataset includes both qualitative and quantitative explanatory variables?
In a classification problem with 0-1 response, is the above method still correct? (In other words, does it make sense to calculate Pearson's correlation coefficient between a binary variable and a quantitative variable?)
What about a multiclass classification problem?

Best Answer

With respect to your question #1, if one goes down the path of using pairwise measures of association as a first step in variable selection, then clearly, Pearson and Spearman correlations aren't appropriate with discrete variables. Identifying a standardized metric that works for both continuous and categorical features becomes the goal. One possibility is to use the magnitude of the value from an F-test as in a one-way ANOVA (see a related link in sidebar to this thread).

However, and as Tim notes in his link, there are many reasons for not using pairwise measures of dependence as a first step in variable selection in multiple regression. These are largely a function of issues related to the conditional nature of model building -- which are not captured in pairwise comparisons -- and the loss of potentially important moderator or interaction effects when applying a rigid cutoff for variable inclusion.

Breiman recommended leveraging the "meta-" output from his random forest routine as input to variable selection. Basically, this involves tracking the performance of a feature in the context of the alternative data and feature landscapes into which it is selected -- a kind of Darwinian approach to variable selection that gets around the absence of linear combinations in pairwise metrics. In addition and depending on how you work it up, it can evaluate moderator or interaction effects...up to the limits of your machine in taking 2-way or higher interactions (the possibilities quickly become enormous, even with a relative handful of features). If you want to get really picky, you can substitute logistic regression as the RF model building framework to deal with the question #2 issue you raise.

That said, among the challenges with variable selection is identifying a "best" methodology. The fact is that every statistician and their brother has an algorithm with a working paper that they use and advocate or, alternatively, algorithms that they disparage. There is wide agreement that stepwise approaches -- which are rooted in significant p-values -- do not deliver "good" answers. There is also wide agreement that Tibshirani's Lasso does provide reasonably good answers, e.g., the NormalDeviate blog described it as one of the top statistical contributions of the last 10 or 20 years. So, if you're looking for an approach to variable selection that easily handles both categorical (including multiclass) and continuous variables and that has wide support in the statistical community, then the Lasso is a safe bet.

My concern has always been with the scalability of any of these statistical solutions, e.g., the Lasso breaks down when faced with a massive number of features or predictors (this isn't a concern of the OPs but I'll discuss it anyway). By "massive" I mean candidate features numbering in the thousands, hundreds of thousands or, I suppose, even millions are possible for some applications. RFs are scalable as a "divide and conquer" method if you have a massively parallel system and are able to run millions of "mini-models" in a finite amount of time.

What are some other similarly scalable solutions? This is an open-ended, rhetorical query.

Related Solutions

Solved – Feature selection by univariate correlation with class labels

You will do it exactly in a same way as with continuous variables. Only your dependent variables have value of 0 or 1. You Pearson's correlation will basically became a t-test

Solved – Feature selection – correlation ranking for tree-based methods

Here's an example that should be in everyone's toolbox regarding regression/decision trees, as it succinctly makes two very important points.

The vertical axis shows a response $y$. There are two predictor variables, $x_1$ ranges from $-1$ to $1$ continuously. The relationship between $y$ and $x_1$ is constructed so that no matter how you partition the $x_1$ axis, the response always averages out to zero. In particular:

> cor(df$x_2, df$y)
[1] -0.001792121

Or, for all intents and purposes, zero.

The other variable $x_2$ allows you to distinguish the "arm" of the $X$ shaped data.

This data has two very interesting features:

The true or correct decision tree for this data splits on $x_2$ first. This allows it to distinguish the arms of the $X$ shape, and immediately breaks the zero correlation structure between $x_1$ and $y$ for all sub-splits.
A greedy algorithm will generally not find the optimal fist split (you could get very lucky, but probably it will find some noisy split on $x_1$)! The first split on $x_2$ does not lead to a reduction in squared error, it is only important because it lets later splits focus on the association between $x_1$ and $y$.

Here is the R code I used to make this plot, so you can experiment yourself

df <- data.frame(
  x_1 = c(seq(-1, 1, length.out=250), seq(1, -1, length.out=250)),
  x_2 = rep(c(1, -1), each=250)
)
df$y <- df$x_1*df$x_2 + rnorm(500, 0, .05)

library(ggplot2)
ggplot(data=df) + geom_point(aes(x=x_1, y=y, color=factor(x_2)))

Best Answer

Related Solutions

Solved – Feature selection by univariate correlation with class labels

Solved – Feature selection – correlation ranking for tree-based methods

Related Question