I have 3 classifier models namely Logistic Regression, Linear-SVM, Decision Trees as an ensemble technique. I am using majority voting as the classifier combination method for prediction. But when I try to calculate area under ROC curve in R, the function roc needs response and predicted probabilities(score). The predicted class labels can be calculated by majority voting technique but how to calculate the score and thus calculate area under roc curve and generate roc curve?
Solved – Generating the ROC curve for ensemble Classifier
classificationensemble learningroc
Related Solutions
Lorenz curve is also known under the name of "lift curve" when applied to classification/ranking. For a given range of predicted probability values, the lift represents a multiplicative increase in the positive class's rate (due to a given predictive model) over a random guess.
rocr package can calculate lift values/curves (the manual also has a concise definition of the lift). The Gini index can be calculated from the area under the lift curve (I typically use cumulative lift value at a given predicted probability threshold instead since it is easier to relate to business metrics)
As I see it, the possibility to refuse classification as "too uncertain" is the whole point of choosing a threshold (as opposed to assigning the class with highest predicted probability).
Of course, you should have some justification for putting the threshold to 0.5: you may also put it up to 0.9 or any other value that is reasonable.
You describe a setup with mutually exclusive classes (closed-world problem). "No class reaches the threshold" can always happen as soon as that threshold is higher than 1/$n_{classes}$, i.e. the same problem occurs in a 2-class problem with threshold, say, 0.9. For threshold = 1/$n_{classes}$ it could happen in theory, but in practice it is highly unlikely.
So your problem is not related (just more pronounced) to the 3-class set-up.
To your second question: you can compute ROC curves for any kind of continuous output scores, they don't even need to claim that they are probabilities. Personally, I don't calibrate, because I don't want to waste another test set on that (I work with very restricted sample sizes). The shape of the ROC anyways won't change.
Answer to your comment: The ROC conceptually belongs to a set-up that in my field is called single-class classification: does a patient have a particular disease or not. From that point of view, you can assign a 10% probability that the patient does have the disease. But this does not imply that with 90% probability he has something defined - the complementary 90% actually belong to a "dummy" class: not having that disease. For some diseases & tests, finding everyone may be so important that you set your working point at a threshold of 0.1. Textbook example where you choose an extreme working point is HIV test in blood donations.
So for constucting the ROC for class A (you'd say: the patient is A positive), you look at class A posterior probabilities only. For binary classification with probability (not A) = 1 - probability (A), you don't need to plot the second ROC as it does not contain any information that is not readily accessible from the first one.
In your 3 class set up you can plot a ROC for each class. Depending on how you choose your threshold, no classification, exactly one class, or more than one class assigned can result. What is sensible depends on your problem. E.g. if the classes are "Hepatitis", "HIV", and "broken arm", then this policy is appropriate as a patient may have none or all of these.
Best Answer
Yes, you need score-type output to compute a receiver-operating-curve. With fixed cutoff, you just get a single point.
However, with an aggregated model, you can use the individual votes as score. I.e. if you aggregate the votes of $n$ classifiers, they form a score ranging from 0 positive to $n$ positive votes in the prediction.
For the aggregated prediction of 3 models, the ROC will only have few points corresponding to the at most $n + 1$ different predicted scores you observe, but that is still a valid ROC.