Solved – Choosing the correct AUC value with RocR package

aucr

I have somewhat of an odd question.

I'm currently working on a dataset and after training my model, I'm trying to get the correct AUC and I'm using the RocR package to get that value.

Im using these commands 

  pred <- predict(model, testData,type='prob')  
  pr=prediction(pred[,2],testData$var)  
  prf <- performance(pr, measure = "tpr", x.measure = "fpr")  

  auc <- performance(pr, measure = "auc")  
  auc <- auc@y.values[[1]]

I've noticed that whether I add the type='prob' changes the value of my auc, meaning that if i predict just the binary output instead of the probabilities, the AUC changes. So my question is which value should I use? I'm a bit lost.

Best Answer

What's your model? Note that predict is not a ROCR function, ROCR starts from prediction. It's your job to specify the model correctly. Of course, what the model generates have a significant impact on you AUC statistics, because you're comparing your prediction against testData$var. You'll need to think what exactly you want to predict. ROC is just a statistical tool, it's garbage in and garbage out, so make sure don't give garbage inputs.

What's testData$var? It's supposed to be the classified labels. Are they related to your binary or probability prediction (I don't know what model you're using)?

Note that you don't choose your AUC statistics, you choose your model that maximize AUC statistics as much as possible.

Related Question