Solved – Features selection using F-score for multiclass classification

classificationfeature selectionmachine learning

I'm going to implement a feature selection algorithm, and I plan to use the F-score for because of its simplicity. The problem is that, the F-score is used for binary classification. How can it be extended for multi-class classification?

My idea is to use one-vs-all method. That is, if I have k classes, I would generate k F-scores for each of these (by taking only one feature at a time and calculate the F-score against all others) and then just take the average. Would this be a good approach?

Best Answer

F-score tends to be very similar to likes of t and Kruskal-Wallis tests, when it comes to feature ranking. So, one solution can be using multi-level alternatives of t-test, e.g., ANOVA or their non-parametric versions.

Related Solutions

Solved – Variable selection procedure for binary classification

A very popular approach is penalized logistic regression, in which one maximizes the sum of the log-likelihood and a penalization term consisting of the L1-norm ("lasso"), L2-norm ("ridge"), a combination of the two ("elastic"), or a penalty associated to groups of variables ("group lasso"). This approach has several advantages:

It has strong theoretical properties, e.g., see this paper by Candes & Plan and close connections to compressed sensing;
It has accessible expositions, e.g., in Elements of Statistical Learning by Friedman-Hastie-Tibshirani (available online);
It has readily available software to fit models. R has the glmnet package which is very fast and works well with pretty large datasets. Python has scikit-learn, which includes L1- and L2-penalized logistic regression;
It works very well in practice, as shown in many application papers in image recognition, signal processing, biometrics, and finance.

Solved – Classification score: SVM

If I understand your question, you can pass the -b flag with option 1 when building the model like this:

../svm-train -b 1 data_file

And then when creating the prediction vector, you do the same:

../svm-precict -b 1 test_file data_file.model output_file

Here is an output_file example that has 3 classes:

labels 0 -1 1
0 0.635655 0.18753 0.176816

Best Answer

Related Solutions

Solved – Variable selection procedure for binary classification

Solved – Classification score: SVM

Related Question