Solved – Kernel logistic regression vs SVM

logisticsvm

As is known to all, SVM can use kernel method to project data points in higher spaces so that points can be separated by a linear space.
But we can also use logistic regression to choose this boundary in the kernel space, so what's the advantages of SVM?
Since SVM uses a sparse model in which only those support vectors make contributions when predicting, does this make SVM faster in prediction?

Best Answer

KLRs and SVMs

  1. Classification performance is almost identical in both cases.
  2. KLR can provide class probabilities whereas SVM is a deterministic classifier.
  3. KLR has a natural extension to multi-class classification whereas in SVM, there are multiple ways to extend it to multi-class classification (and it is still an area of research whether there is a version which has provably superior qualities over the others).
  4. Surprisingly or unsurprisingly, KLR also has optimal margin properties that the SVMs enjoy (well in the limit at least)!

Looking at the above it almost feels like kernel logistic regression is what you should be using. However, there are certain advantages that SVMs enjoy

  1. KLR is computationally more expensive than SVM - $O(N^3)$ vs $O(N^2k)$ where $k$ is the number of support vectors.
  2. The classifier in SVM is designed such that it is defined only in terms of the support vectors, whereas in KLR, the classifier is defined over all the points and not just the support vectors. This allows SVMs to enjoy some natural speed-ups (in terms of efficient code-writing) that is hard to achieve for KLR.
Related Question