Say we have multiple SVMs used in a one-vs-all approach, such that classes a, b, c correspond to 3 SVMs trained positively on the class and then negatively on all other classes.
When we test these we get 3 results. For a class a sample we may get a:0.9, b:0.2, c:0.3.
Because each SVM is computed slightly differently and has different scales, I cannot compare them. Is there an approach that will work on the outputs of one-vs-all classifiers to calibrate the outputs?
I have found a paper which discusses it in section 2.1 (http://www.metarecognition.com/wp-content/uploads/2012/05/cvpr2012_attributes.pdf) but I am unsure if this would work one my one-vs-all approach as the paper seems to be talking about applying this to only one classifier and I have 3.
Best Answer
Proposed Solution: Calibration
I have read the "Multi-class" part of your post, but since you are using One vs. All SVMs, i think you should reconsider solving the problem at the binary level. You could calibrate the single svms, so that the resulting output values are comparable.
Calibration methods for the binary SVMs (so that means also applicable in the one vs. all scenario) are Platt scaling1 and Isontonic regression. A nice overview with python code is available here.
For your own use case you would then calibrate each OvA SVM separately and afterwards the calibrated outputs for
a,bandcshould be comparable.What does calibration do here?
The key thing here is, that SVMs themselves, are not probabilistic. The output value you mentioned is usually a function of the classified point's distance to the hyperplane. So we are using a heuristic which has no further significance. The goal of this heuristic is that higher numbers are more likely to be the correct result.
You can measure the signifance of your output values using a reliability plot. I will cut short here but essentially you want your reliability curve to be as close as possible to the diagonal. The calibration adds another mapping of output values to calibrated output values. This can handle for example classifiers which have a bias towards high output values. Think of it as another translation step "Ok i got that really confident
0.9from you classifier A, but i know you always are over-confident so let's make this a0.5". So a0.5value of classifier A should be closer to a0.5value of classifier B in the end.Keep in mind, when using calibration you have to work thoroughly as usual (train/dev/test set).
1. Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3), 61-74.