Solved – Differences in AUC calculation between pROC and ROCR

aucmachine learningrroc

Does anyone know the difference in calculation between these two AUC packages? They get different results when I add in positives with predicted value of 0 (simulating a prob model where many outputs will be zero e.g. randomForest). So I would like to know why there is the kink in the pROC results (red) and the AUC 'flips'. The plot below illustrates this behaviour.

Also, this stems from the problem of having to compute AUC's on models with many positives assigned probabilities of zero.

 library(ROCR)
 library(pROC)

  start.ds <- data.frame(response=gl(2, 1000, labels = c("1", "0")),pred=ifelse(as.numeric(gl(2, 1000, labels = c("0", "1")))==2,0,1))
  add.positives <- data.frame(response=gl(1, 1000, labels = c("1")),pred=0)
  add.positives <- rbind(add.positives, add.positives)

  nadd <- nrow(add.positives)
  theSeq <- seq(1, nadd, 10)
  allAUCs <- NULL
  for(i in theSeq){
       print(i)
       temp.ds <- rbind(start.ds,add.positives[1:i,])
       temp.ds <- temp.ds[order(temp.ds$pred,decreasing=TRUE),]

       aucPROC <- as.numeric(auc(temp.ds[,1],temp.ds[,2]))
       aucAUCPackage <- performance(prediction(temp.ds[,2],temp.ds[,1]),"auc")@y.values[[1]]
       allAUCs <- rbind(allAUCs, data.frame(obs=i, aucPROC=aucPROC, aucAUCPackage=aucAUCPackage))

    }

  plot(allAUCs$aucPROC,col="red",type="l")
  lines(allAUCs$aucAUCPackage,col="black",type="l")

enter image description here

Best Answer

You never specified what is a positive result. Try the following:

aucPROC <- as.numeric(auc(temp.ds[,1],temp.ds[,2], direction="auto"))
aucPROC <- as.numeric(auc(temp.ds[,1],temp.ds[,2], direction="<"))
aucPROC <- as.numeric(auc(temp.ds[,1],temp.ds[,2], direction=">"))

Also you need to make sure that positive and negative examples are assigned to the correct values, although with 0s and 1s it will probably be detected correctly...

aucPROC <- as.numeric(auc(temp.ds[,1],temp.ds[,2], levels = c(0, 1), direction="<"))

See ?roc for more details.

Related Solutions

AUC-Value – How to Calculate AUC Value from ROC Curve

Edit: since you apparently do have scores and actual outcomes, you can calculate it. One tool that can do the job would be the pROC package in R. It contains an AUC function that takes as arguments the predicted scores and actual outcomes. Have a look at its documentation http://cran.r-project.org/web/packages/pROC/index.html

-- INITIAL ANSWER --
There is no such tool, because you lack necessary information. You need to have a score for each prediction as well as its true outcome. Without that kind of information, it is impossible to calculate AUC.

Solved – Differences in AUC calculation in R between pROC and AUC

First, notice that $1 - 0.4942452 = 0.5057548 ≈ 0.5058$.

Second, notice that ?AUC::roc accepts only labels as {0,1}, with predictions being the probability of the positive event.

predictions: A numeric vector of classification probabilities
      (confidences, scores) of the positive event.

labels: A factor of observed class labels (responses) with the only
      allowed values {0,1}.

pROC::roc makes no such assumptions on the input data. It has some heuristics to detect which level is the positive and negative case, and more importantly if value of the predictions score are higher in the positive or negative cases. They don't have to be probabilities, any numeric values or even ordered factors will do. This can be controlled by the levels and direction arguments:

levels: the value of the response for controls and cases
      respectively. By default, the first two values of
      ‘levels(as.factor(response))’ are taken, and the remaining

direction: in which direction to make the comparison?  “auto”
      (default): automatically define in which group the median is
      higher and take the direction accordingly.  “>”: if the
      predictor values for the control group are higher than the
      values of the case group (controls > t >= cases).  “<”: if
      the predictor values for the control group are lower or equal
      than the values of the case group (controls < t <= cases).

pROC auto-detects the direction, based on the median values in the two groups that are being compared.

Please note that when you sample curves with AUC close to 0.5, this will result in a bias towards a higher AUC; therefore you should keep the direction fixed. To get results identical to AUC::roc you should use the following:

x <- pROC::auc(outcome, predictor, levels = c(0, 1), direction = "<")

Best Answer

Related Solutions

AUC-Value – How to Calculate AUC Value from ROC Curve

Solved – Differences in AUC calculation in R between pROC and AUC

Related Question