The problem of designing a multi-class classifier using LDA can be expressed as a 2 class problem(one vs everything else) or a multi-class problem.
Why is it that in certain cases Multi-class LDA classifier out-performs 2 class LDA (one vs everything else) or vice-versa.
Best Answer
I think that Multi-class LDA classifier always (well, in most practical tasks) out-performs 2 class LDA. And I will try to describe why.
Have a look at the example dataset:
You have three classes here. And let's say you want to build one-vs-other classifier with LDA for the blue class.
The estimated mean for class "blue" is zero, but the estimated mean for class "other" is zero as well. And the covariance is the same from the definition of LDA. That means that LDA will respond with the label that has more elements. And it will never return class "blue" at all!
For Multi-class LDA it will manage to find the right classes perfectly.
The background about this is that the mixture of Gaussians is not a Gaussian any more in most cases. So this assumption of LDA fails. And I must say it is really difficult to come up with an example of a dataset where every class is Gaussian, and they still Gaussian after we join them.
That's why I would highly recommend to use Multi-class LDA. Hope it will help!