Solved – Feature Selection Packages in R, which do both regression and classification

feature selectionr

I am very new to R. I am learning machine learning right now.
Very sorry, if this question appears to be very basic.
I am trying to find a good feature selection package in R.
I went through Boruta package. It is a good package but I read that it is only useful for classification.

I want to do implement feature selection in R for regression tasks. I went through the caret package documentation but for my level, it is very difficult to understand.

Can any one please point me to a good tutorial or list any good packages or most frequently used packages in R for feature selection.

Any help would be appreciated.
Thanks in advance.

Best Answer

You can also have a look at FSelector, varSelRF. FSelector contains multiple functions for feature selection based for example on the chi square test, on the information theory (entropy, mutual information, gain ratio,...), on the correlation between feature, consistency etc... varSelRF is a useful package for feature selection using random forests with backwards variable elimination and with importance spectrum.

Related Solutions

Solved – Feature selection + classification in Caret

You should be able to accomplish everything you want with the sbf function instead. I originally assumed it worked the same way you are, but the functionality given by sbf is apparently more like a super set of what's available in train.

For example, something like this sounds like what you're getting at:

fit <- sbf(
  form = response ~ .,
  data = d, method = "glmnet", 
  tuneGrid=expand.grid(.alpha = .01, .lambda = .1),
  preProc = c("center", "scale"),
  trControl = trainControl(method = "none"),
  sbfControl = sbfControl(functions = caretSBF, method = 'cv', number = 10) 
)

This would run 10 outer folds and fit a single glmnet model to each, using only a feature subset. You could also specify some number of cv folds for trControl and a parameter grid to do training on inner folds.

Feature Selection in Classification – Why Is It Important?

Your intuition is quite correct. In most situations, feature selection represents a desire for simple explanation that results from three misunderstandings:

The analyst does not realize that the set of "selected" features is quite unstable, i.e., non-robust, and that the process of selection when done on another dataset will result in a quite different set of features. The data often do not possess the information content needed to select the "right" features. This problem gets worse if co-linearities are present.
Pathways, mechanisms, and processes are complex in uncontrolled experiments; human behavior and nature are complex and not parsimoneous.
Predictive accuracy is harmed by asking the data to tell you both what are the important features and what are the relationships with $Y$ for the "important" ones. It is better to "use a little bit of each variable" than to use all of some variables and none for others (i.e., to use shrinkage/penalization).

Some ways to study this:

Do more comparisons of predictive accuracy between the lasso, elastic net, and a standard quadratic penalty (ridge regression)
Bootstrap variable importance measures from a random forest and check their stability
Compute bootstrap confidence intervals on ranks of potential features, e.g., on the ranks of partial $\chi^2$ tests of association (or of things like univariate Spearman $\rho$ or Somers' $D_{xy}$) and see that these confidence intervals are extremely wide, directly informing you of the difficulty of the task. My course notes linked from http://biostat.mc.vanderbilt.edu/rms have an example of bootstrapping rank order of predictors using OLS.

All of this applies to both classification and the more general and useful concept of prediction.

Best Answer

Related Solutions

Solved – Feature selection + classification in Caret

Feature Selection in Classification – Why Is It Important?

Related Question