SVM – Why Scaling Features Can Decrease SVM Performance

svm

I have used scaling on features of a model which contains 40 features (all columns are numbers) and a binary output variable.

This is the Kaggle contest here I've scaled the features assuming it would deliver better performance, but with a rbf kernel SVM, the accuracy with 10 fold CV fell from 0.92 to 0.87

Here is a box plot of features before and after scaling:

enter image description here
enter image description here

What I would like to know is why scaling decreases classifier performance? I have not seen any discussions that point at this type of outcome.

Best Answer

The problem is that you used the default parameter values in both cases. Apparently, the default values happened to be better for your data set before scaling (this is a coincidence).

When using SVM, the parameters $c$ and $\gamma$ play a crucial role and it is your task to find the best values. Your intuition is correct: the optimal performance is better when all features are scaled properly (or at least 99.99% of the time). Unfortunately, neither of your settings had optimal parameters which led to a result that seemed to reject your intuition.

Searching the optimal values for $c$ and $\gamma$ is typically done via a grid search (e.g. search a set of $<c,\gamma>$ combinations). You can estimate the performance of an SVM for a given set of parameters using cross-validation.

In pseudo-code, the general idea is this:

for c in {set of possible c values}
    for gamma in {set of possible gamma values}
        perform k-fold cross-validation to find accuracy
    end
end
train svm model on full training set with best c,gamma-pair

You can find a good beginner's tutorial here.

Related Question