SVM – Where to Read About Gamma Coefficient in SVM in Scikit-Learn?

scikit learnsvm

Scikit learn support vector machine algorithm have a couple of coefficients which meaning I can not understand.

gamma : float, optional (default=0.0)

Kernel coefficient for ‘rbf’,
‘poly’ and ‘sigmoid’. If gamma is 0.0 then 1/n_features will be used
instead.

and

coef0 : float, optional (default=0.0)

Independent term in kernel function. It is only significant in ‘poly’
and ‘sigmoid’.

I tried to google for gamma in support vector machines, but have not found anything relevant. Can anyone explain me what is the meaning of these parameters and reasonable value for them.

Best Answer

The RBF kernel function is as follows, for two vectors $\mathbf{u}$ and $\mathbf{v}$: $$ \kappa(\mathbf{u},\mathbf{v}) = \exp(-\gamma \|\mathbf{u}-\mathbf{v}\|^2). $$ The hyperparameter $\gamma$ is used to configure the sensitivity to differences in feature vectors, which in turn depends on various things such as input space dimensionality and feature normalization.

If you set $\gamma$ too large, you will end up overfitting. In the limit case $\gamma\rightarrow\infty$, the kernel matrix becomes the unit matrix which leads to a perfect fit of the training data, though an entirely useless model.

The optimal value of $\gamma$ depends entirely on your data, any rules of thumb should be taken with a pound of salt. That said, you can use specialized libraries to optimize hyperparameters for you (e.g. Optunity (*)), in the case of SVM with RBF kernel that is $\gamma$ and $C$. You can find an example of optimizing these parameters automatically with Optunity and scikit-learn here.

(*) disclaimer: I'm the lead developer of Optunity.