Solved – What value of ANOVA F-Value is significant

anovafeature selectionscikit learn

I'm using ANOVA F-Value from Scikit-learn (link) to find importance of features in a machine learning task. I have numeric feature and binary classification. I'm also filtering the features whose p-value is > 0.05, but I don't know what value for the ANOVA F-Value is good enough?

Best Answer

I'm also filtering the features whose p-value is > 0.05

Don't do that. $p$-values of coefficients are a bad way to select models or features because they don't directly quantify any useful measures of model quality. Besides, while obtaining $p < α$ tells you that a null hypothesis is infeasible, obtaining $p > α$ doesn't much support its feasibility, so when a test fails to reject the null hypothesis, it doesn't tell you anything.

Related Solutions

Solved – SciKit Learn get feature importance for multiclass classification using Decision Tree

You could modify your problem by using multiple one-vs-rest classifiers. For example train a classifier to distinguish between (1) class_a and (2) rest. Then you can access the feature importance. Nonetheless, the feature importance is not the importance of a feature for a certain class, but a measure for the usability of a single feature to distinguish two classes (here one-vs-rest).

Therefore your the feature importance attribute does not answer the question "Which feature is important for a class?" but rather answers the question "Which feature helps best at distinguishing class_a from other classes present".

Solved – How to correctly interpret f-regression values during feature selection

Pick the variables with the highest F-statistic. If you want, you can use a more handy function, like feature_selection.SelectPercentile.

selector = feature_selection.SelectPercentile(feature_selection.f_regression, percentile=30)
selector.fit(X_train, y_train)
selector.get_support(True)

The method get_support (with True) will return the indexes of the selected features, according to the percentile.

Best Answer

Related Solutions

Solved – SciKit Learn get feature importance for multiclass classification using Decision Tree

Solved – How to correctly interpret f-regression values during feature selection

Related Question