Solved – Comparing two Machine learning Models using ROC curves

machine learningnaive bayesrocsvm

I have developed an SVM-Model using x data. ROC curve was generated using 5-fold cross-validation. Now I want to compare my new SVM-model with a published Bayes-classifier. I have got the predictions for x data, using both published model and the SVM-model.

Data looks like:

True-Label, Predicted labels

Item True-Label Model-1 Model-2
 AFA  1          0      1
 AFB  0          0      0
 AFD  0          1      1
 BFA  1          1      1
 .....
 .....
 .....
 ZFD  0          1       0

1) Can an ROC Curve be generated using above data, if yes, how to generate one? Will it be appropriate to compare these curves with each other?

2) What are the other approaches to compare these kind of models(I have calculated metrics like sensitivity, specificity,…).

EDIT-1

For the published model I have TPR and FPR values (not for same data). Bayes classifier also outputs probability score along with label. How can I use it for generating Curves?

EDIT-2
Data generated from prediction by Bayes Classifier Contains Probability Values.
I Formatted the output into following table (for x data). 

True-Label Predicted-Label Probability
0          0               0.27
0          1               0.97
1          1               0.6
1          1               0.978

Can an ROC curve be generated using this data ?

Best Answer

As Marc Claesen points out, some kind of certainty measure is needed. Below I have showed two approaches of how to form ROC curves.

  1. If the classifier can output a probabilistic measure, such one can be used in e.g. 5-fold cross validation to form a ROC plot.
  2. If the classifier only outputs predicted labels, then the certainty of predictions can be estimated with bagging. The training set is bootstrapped and modeled e.g. 100 times and the cross validated out-of-bag predictions are used for ROC curves.

(1,probabilistic svm, black curve) and (2,bagged svm, red curve) enter image description here

For multi-class ROC curves use e.g. "1 vs. rest" method, check out this post

rm(list=ls())
set.seed(1)
library(e1071)
library(AUC)

data(iris)
iris = iris[1:100,] #remove one species, to simplify to a 2-class problem
iris[1:4] = lapply(iris[1:4],jitter,amount=2) #add noise, otherwise too easy
#NB ROC PLOT will change for each new random noise component (jitter)
X = iris[1:100,names(iris)!="Species"]
y = iris[1:100,"Species"]

#cross-validated SVM-probability plot
folds = 5
test.fold = split(sample(1:length(y)),1:folds) #ignore warning
all.pred.tables = lapply(1:folds,function(i) {
  test = test.fold[[i]]
  Xtrain = X[-test,]
  ytrain = y[-test ]
  sm = svm(Xtrain,ytrain,prob=T) #some tuning may be needed
  prob.benign = attr(predict(sm,X[test,],prob=T),"probabilities")[,2]
  data.frame(ytest=y[test],ypred=prob.benign) #returning this
})
full.pred.table = do.call(rbind,all.pred.tables)
plot(roc(full.pred.table[,2],full.pred.table[,1]))


#bagged OOB-cross validated SVM AUC plot
n.bootstraps=100 #how many models to train
inbag.matrix = replicate(n.bootstraps,sample(1:length(y),replace=T))
all.preds = sapply(1:n.bootstraps,function(i) {
  inbag = inbag.matrix[,i]
  outOfBag = which(!1:length(y) %in% inbag)
  Xtrain = X[inbag,]
  ytrain = y[inbag ]
  sm = svm(Xtrain,ytrain) #some tuning may be needed
  pred.label = rep(NA,length(y))
  pred = predict(sm,X[outOfBag,])
  pred.label[outOfBag] = levels(pred)[as.numeric(pred)]
  addNA(factor(pred.label))
})

bag.prob = apply(all.preds,1,function(aRow){
  inbag = which(is.na(aRow))
  mean(aRow[-inbag] == levels(y)[2])
})
plot(roc(bag.prob,y),col="red",add=TRUE)
Related Question