I'm working on a multi-class problem which I have redefined as a series of binary problems (i.e. a one vs all classification problem). However, each observation can belong to more than one class. For example, if my observations where different kinds of fruit my classes might represent different characteristics such red and round. In some cases a fruit is both red and round.
My question is: what should I consider when evaluating my binary models? Can one simply use metrics such as accuracy to understand the performance of the model. If I have three different classes (i.e red, round and sweet) is it acceptable to merely take the mean accuracy of the three binary classification tasks as the accuracy of my model as a whole?
This is a little different than most multi-class classification problems I've seen where all the classes are independent.
Best Answer
Since each sample might belong to some classes, I would have break in to binary classes not in a one vs. all way but look for membership in the class (e.g., IsRed? IsRound?).
As for the evaluation, you should use measures that fit your needs. I'm not familiar with your needs but it is quite rare that that mean of accuracies will fit the need.
You can either try to evaluate each classifier separately (accuracy of IsRed, accuracy of IsRound) or the accuracy of a full prediction (All relevant classifiers correctly classified the sample).