I am recently learning machine learning and got to know about feature selection.
I am wondering if a wrapper method like "Recursive feature elimination" provided by scikit tests cross validation on all subset of features, isn't it the best way to select features compared to other feature selection methods. Such as, Removing features with low variance.
Because, "Recursive feature elimination" checks all subset of features in the training set.
If not, please explain the reason.
Best Answer
I think you are confusing RFE with search algorithms that search over all possible subsets of features. RFE does something similar, but it does not check all possible combinations. At every step, RFE simply eliminates a certain number or a certain percentage of the lowest ranking features in your model, and retrains. This makes the assumption that these features are not important to begin with, so they can be eliminated. It then continues eliminating features as such, until your stop criterion is reached. Depending on your problem, the resulting features may or may not be the same ones you would obtain if you searched over all combinations of features. Most likely, they are not.
For many real world problems searching over all possible subsets of features is intractable. RFE becomes a good compromise.