kernel trickmachine learningsvm
I am carrying out SVM and was interested in knowing the parameters that could be varied for each kernel. I am using 3 kernels: RBF, linear and polynominal.
These are the parameters that i think can be changed:
For Linear : C
For RBF: C and gamma
For Polynominal : C and gamma
Would like to hear some feedback whether i am wrong or correct. If there are any other parameters, do tell me… Thanks.
Check out A practical guide to SVM Classification for some pointers, particularly page 5.
We recommend a "grid-search" on $C$ and $\gamma$ using cross-validation. Various pairs of $(C,\gamma)$ values are tried and the one with the best cross-validation accuracy is picked. We found that trying exponentially growing sequences of $C$ and $\gamma$ is a practical method to identify good parameters (for example, $C = 2^{-5},2^{-3},\ldots,2^{15};\gamma = 2^{-15},2^{-13},\ldots,2^{3}$).
Remember to normalize your data first and if you can, gather more data because from the looks of it, your problem might be heavily underdetermined.
The linear kernel is what you would expect, a linear model. I believe that the polynomial kernel is similar, but the boundary is of some defined but arbitrary order
(e.g. order 3: $ a= b_1 + b_2 \cdot X + b_3 \cdot X^2 + b_4 \cdot X^3$).
RBF uses normal curves around the data points, and sums these so that the decision boundary can be defined by a type of topology condition such as curves where the sum is above a value of 0.5. (see this picture )
I am not certain what the sigmoid kernel is, unless it is similar to the logistic regression model where a logistic function is used to define curves according to where the logistic value is greater than some value (modeling probability), such as 0.5 like the normal case.
Best Answer
C is the slack parameter for SVM, regardless of kernel chosen, it has nothing to do with the kernel. It determines the tradeoff between a wide margin and classifier error. Large C means you are allowing little slack and the model will fit tighter to your data, small C means you are allowing a lot of slack and the model will have more error on the training set, but be less sensitive to noise.
Linear Kernel: No parameters
Polynomial: (gamma*u'*v + coef0)^degree (using libsvm's nomenclature) Degree is the main parameter here, but you can also vary gamma & coef0 to make the kernel non-symmetric. If you are just starting out, I recommend you stick to varying degree.
RBF: exp(-gamma*|u-v|^2) Gamma is the main parameter.