Solved – Measuring accuracy of a logistic regression-based model

logisticr-squaredregression

I have a trained logistic regression model that I am applying to a testing data set. The dependent variable is binary (boolean). For each sample in the testing data set, I apply the logistic regression model to generates a % probability that the dependent variable will be true. Then I record whether the acutal value was true or false. I'm trying to calculate an $R^2$ or Adjusted $R^2$ figure as in a linear regression model.

This gives me a record for each sample in the testing set like:

prob_value_is_true         acutal_value
   .34                          0
   .45                          1
   .11                          0
   .84                          0
    ....                        ....

I am wondering how to test the accuracy of the model. My first attempt was to use a contingency table and say "if prob_value_is_true > 0.80, guess that the actual value is true" and then measure the ratio of correct to incorrect classifications. But I don't like that, because it feels more like I'm just evaluating the 0.80 as a boundary, not the accuracy of the model as a whole and at all prob_value_is_true values.

Then I tried to just look at each prob_value_is_true discrete value, as an example, looking at all samples where prob_value_is_true=0.34 and measuring the % of those samples where the acutal value is true (in this case, perfect accuracy would be if the % of samples that was true = 34%). I might create a model accuracy score by summing the difference at each discrete value of prob_value_is_true. But sample sizes are a huge concern here, especially for the extremes (nearing 0% or 100%), such that the averages of the acutal values are not accurate, so using them to measure the model accuracy doesn't seem right.

I even tried creating huge ranges to ensure sufficient sample sizes (0-.25, .25-.50, .50-.75, .75-1.0), but how to measure "goodness" of that % of actual value stumps me. Say all samples where prob_value_is_true is between 0.25 and 0.50 have an average acutal_value of 0.45. Is that good since its in the range? Bad since its not near 37.5% (the center of the range)?

So I'm stuck at what seems like should be an easy question, and hoping someone can point me to a resource or method to calculate an accuracy stastic for a logistic regression model.

Best Answer

A measure that is often used to validate logistic regression, is the AUC of the ROC curve (plot of sensitivity against 1-specificity - just google for the terms if needed). This, in essence, evaluates the whole range of threshold values.

On the downside: evaluating the whole range of threshold values may be not what you're after, since this (typically) includes thresholds that result in very large numbers of false negatives or false positives. There are versions of the AUC that account for this (partial AUC), so if that is an issue for you, you can look into that.

Related Question