Machine Learning – Using Correlation Based Feature Selection (CFS) Tool

correlationfeature selectionmachine learning

Is there any tool or script that was implemented for correlation based feature selection? My feature vector data is in a large-scaled data file, so if I use tools like Weka for feature selection, I don't get any result!

Best Answer

You could use findCorrelation from R caret to achieve this, which selects the optimal subset of features to minimize their inter-feature correlation below a specified threshold. You will have to try out if this still works for your data (it will also run into problems with very huge dataset I assume):

# original correlation
library(corrplot)
corrplot(cor(mtcars), type = 'lower')

# finding optimal feature subset (as features to be removed)
library(caret)
toRemove <- findCorrelation(cor(mtcars), cutoff = 0.7)

# remaining correlation
corrplot(cor(mtcars[,-toRemove]), type = 'lower')

Related Solutions

Solved – Step-wise feature selection with caret

caret has a stepLDA method available in train:

slda <- train(Species ~ ., data = iris,
              method = "stepLDA",
              trControl = trainControl(method = "cv"))

This uses stepclass in the klaR package. There are also LDA feature selection tools in caret using rfe and sbf that would be helpful.

Max

Feature Selection – Information Gain vs. Mutual Information for Feature Selection

They are identically the same.

If $X$ is a nominal feature with $k$ different values and $C$ is your target class with $m$ classes.

\begin{align} \mbox{MI} &= \sum_{i=1}^k\sum_{j=1}^m P(x_i,c_j)\log \frac{P(x_i,c_j)}{P(x_i)P(c_j)}\\ &=-\sum_{i=1}^k\sum_{j=1}^m P(x_i,c_j)\log P(c_j) + \sum_{i=1}^k\sum_{j=1}^m P(c_j|x_i)P(x_i)\log P(c_j|x_i)\\ &= -\sum_{j=1}^m P(c_j)\log P(c_j) + \sum_{i=1}^kP(x_i)\sum_{j=1}^m P(c_j|x_i)\log P(c_j|x_i)\\ &= H(C) - H(C|X) = \mbox{IG} \end{align}

This is a case of inconsistent naming. You might want to have a look at this question too.

Best Answer

Related Solutions

Solved – Step-wise feature selection with caret

Feature Selection – Information Gain vs. Mutual Information for Feature Selection

Related Question