Solved – AUC for test data less than training data

auclogisticmachine learning

I split the training data into a train (75%) and test data (25%). I got an AUC for test data less than training data. does this mean overfitting ?

Best Answer

It's hard to detect overfitting with just one data point. Overfitting happens when the performance on the training set is improved but the performance on the test data gets worse. What you got is consistent with overfitting but it could also be due to some other random difference between the training and test data.

Related Solutions

Solved – R AUC never less than 0.5

There is no auc() function in the randomForest package. But based on the argument names you used (obs and pred), I think you might have used the auc() function in the SDMTools package. And yes, this function does flip the results if the calculated AUC is less than 0.5:

> SDMTools::auc
function (obs, pred) 
{
    … code to calculate the AUC …
    if (AUC < 0.5) 
        AUC = 1 - AUC
    return(AUC)
}

This might be seen as a nice convenience feature (it’s easy to forget exactly how AUC functions want their arguments coded, and if you get an AUC < .5 in real life, you have usually just used the inverse/incorrect coding of the response vector ), but I think it’s a bad idea. If you try models that are (almost) as bad as random, e.g., in simulations, you will get estimates that are biased high (compared the ‘correct’ AUC estimator).

If you want the correct AUC estimates, you can use either

ROC() from the Epi packages. It draws a nice ROC plot, with the AUC embedded, and also returns an object with the AUC stored as the AUC element.
rcorr.sens() from the rms package. It returns a list with the AUC stored as the first element.
roc() from the pROC package if you manually specify the direction argument. It returns an object with the AUC stored as the auc element.

Machine Learning – Why AUC Score is Less Than 0.5 for Logistic Regression?

UPDATE: Sycorax posted the following link in the comments: Can a random forest be used for feature selection in multiple linear regression? deals with this problem and describes why this might not work too well.

Similar explanation: your data/model might suffer from the Curse of dimensionality, as logistic regression is prone to fall to this curse.

Several points: (might be comments with enough reputation)

pipe.fit(X_train, y_train)

Where did you define the training data?

Have you tried class_weight="balanced" for logistic regression? This might produce a different rate of misclassification.

What were the results without the RFE step?

Best Answer

Related Solutions

Solved – R AUC never less than 0.5

Machine Learning – Why AUC Score is Less Than 0.5 for Logistic Regression?

Related Question