classificationmachine learningrocunbalanced-classes
Thanks in advance for the help.
I have an unbalanced dataset that I am using for a binary classification problem. The classes are unbalanced. I believe that in such a case that weighted area under roc (receiver operating curve) is the proper way to analyze my results. However, I don't full understand the difference between area under roc and weighted area under roc.
AUC (based on ROC) and overall accuracy seems not the same concept.
Overall accuracy is based on one specific cutpoint, while ROC tries all of the cutpoint and plots the sensitivity and specificity. So when we compare the overall accuracy, we are comparing the accuracy based on some cutpoint. The overall accuracy varies from different cutpoint.
Suppose you're estimating the posterior probabilities of class membership for a binary problem. I'll denote the two classes by Group 1 and Group 2.
Now suppose that we fit a model and on a holdout set we get the following histogram of our predicted probabilities where each point in the histogram corresponds to $\hat P(y_i = \textrm{Group 1} | x_i)$:
The reddish histogram corresponds to the probabilities for observations that truly belong to group 1, while the blue histogram is for observations that truly belong to group 2. In this case we have perfectly separated the two classes so we'll get an AUC of 1. But if we threshold our probabilities at 1/2 (not necessarily the best thing to do but this is common) we'll misclassify half of this set by labeling everything as Group 2.
Based on what you've described it seems like this sort of thing could be happening to you.
Best Answer
One of the advantages to ROC curves is that they are agnostic to class skew. ROC curves remain the same whether your data is balanced or not, bar some finite-sample effects when you have very few examples of one class.
As such, weighted ROC curves have nothing to do with class balance. Instead, weighted ROC curves are used when you're interested in performance in a certain region of ROC space (e.g. high recall) and was proposed as an improvement over partial AUC (which does exactly this but has some issues). You can read more about it in Weighted Area Under the Receiver Operating Characteristic Curve and Its Application to Gene Selection by Li and Fine.