In the scenario that I have a binary classification problem, and use a binary classifier to train and test my model, assuming everything else is constant, would using a multi-class classifier with 2 labels (one vs. all) produce the same results? (I know I should use the binary classifier, but I'm simply curious).
Solved – Binary Classification vs Multi-class Classification
classificationlogisticmachine learningmulti-class
Related Solutions
I think that Multi-class LDA classifier always (well, in most practical tasks) out-performs 2 class LDA. And I will try to describe why.
Have a look at the example dataset:
You have three classes here. And let's say you want to build one-vs-other classifier with LDA for the blue class.
The estimated mean for class "blue" is zero, but the estimated mean for class "other" is zero as well. And the covariance is the same from the definition of LDA. That means that LDA will respond with the label that has more elements. And it will never return class "blue" at all!
For Multi-class LDA it will manage to find the right classes perfectly.
The background about this is that the mixture of Gaussians is not a Gaussian any more in most cases. So this assumption of LDA fails. And I must say it is really difficult to come up with an example of a dataset where every class is Gaussian, and they still Gaussian after we join them.
That's why I would highly recommend to use Multi-class LDA. Hope it will help!
The softmax function gives a proper probability for each of the possible classes:
$$
P(y=j|x,\{w_k\}_{k=1...K}) = \frac{e^{x^\top w_j}}{\sum_{k=1}^K e^{x^\top w_k}}
$$
This is nice if you want to interpret your classification problem in a probabilistic setting. Benefits of using the probabilistic formulation include being able to place priors on the parameters and obtaining a posterior distribution over classes.
That said, maybe you can imagine a really good classifier that isn't of this form. Perhaps it is of a form that is generally difficult to express (e.g. SVM -- here for multi-class details). If some such complicated classifier works well for you on a given task, perhaps you don't want to use the [potentially weaker] softmax classifier. In such a setting, there may not be a clear all-way output, so you have to settle for repeated one-vs-others classification schemes.
One more counterpoint...you could also augment the expressive power of the softmax-style approach by changing the input to the exponential. For example, it would be straightforward to replace each linear component $x^\top w_j$ with a quadratic expression $x^\top w_j + x^\top A_j x$. Other such augmentations are conceivable.
Related Question
- Solved – Evaluation of binary approach to one vs all multi-class classification
- Solved – Multi-class classification easier than binary classification
- Solved – Improve performance for weak class in multi-class classification
- Solved – How does Binary Relevance work on multi-class multi-label problems
- Classification – Multi-Label vs. Multi-Class Classification: Differences and Uses
Best Answer
"Binary classification" is simply multi-class classification with 2 labels. However, several classification algorithms are designed specifically for the 2-class problem, where the response is modeled as the outcome of a Bernouilli trial.
Of course, this can be generalized to the n-dimensional setting by modeling as the outcome of a multinomial sample. For instance logistic regression can be generalized to multinomial logistic regression.
Some models, such as decision trees and ensembles of trees, can take a different form however when applied to n-class classification.
So the answer is highly dependent on the implementation of the algorithm you're using. But the baseline answer is that any multiclass classification algorithms can be used for 2-class classification, and most 2-class classification algorithms can be generalized for multi-class classification.