According to the documentation of the StandardScaler object in scikit-learn:
For instance many elements used in the objective function of
a learning algorithm (such as the RBF kernel of Support Vector
Machines or the L1 and L2 regularizers of linear models) assume that
all features are centered around 0 and have variance in the same
order. If a feature has a variance that is orders of magnitude larger
that others, it might dominate the objective function and make the
estimator unable to learn from other features correctly as expected.
I should scale my features before classification. Is there any easy way to show why I should do this? References to scientific articles would be even better. I already found one but there are probably many other.
Best Answer
All kernel methods are based on distance. The RBF kernel function is $\kappa(\mathbf{u},\mathbf{v}) = \exp(-\|\mathbf{u}-\mathbf{v}\|^2)$ (using $\gamma=1$ for simplicity).
Given 3 feature vectors: $$ \mathbf{x}_1 = [1000, 1, 2], \quad \mathbf{x}_2 = [900, 1, 2], \quad \mathbf{x}_3 = [1050, -10, 20]. $$
then $\kappa( \mathbf{x}_1, \mathbf{x}_2) = \exp(-10000) \ll \kappa(\mathbf{x}_1, \mathbf{x}_3) = \exp(-2905)$, that is $\mathbf{x}_1$ is supposedly more similar to $\mathbf{x}_3$ then to $\mathbf{x}_2$.
The relative differences between $\mathbf{x}_1$ and: $$ \mathbf{x}_2 \rightarrow [0.1, 0, 0],\quad \mathbf{x}_3 \rightarrow [0.05, -10, 10]. $$
So without scaling, we conclude that $\mathbf{x}_1$ is more similar to $\mathbf{x}_3$ than to $\mathbf{x}_2$, even though the relative differences per feature between $\mathbf{x}_1$ and $\mathbf{x}_3$ are much larger than those of $\mathbf{x}_1$ and $\mathbf{x}_2$.
In other words, if you do not scale all features to comparable ranges, the features with the largest range will completely dominate in the computation of the kernel matrix.
You can find simple examples to illustrate this in the following paper: A Practical Guide to Support Vector Classification (Section 2.2).