(I am following this paper, from page 47 on http://www.bundesbank.de/Redaktion/EN/Downloads/Tasks/Banking_supervision/working_paper_no_14_studies_on_the_validation_of_internal_rating_systems.pdf?__blob=publicationFile)
I have some model from which I can construct ROC and calculate its $AUC$. 95% confidence interval will be $[AUC – x, AUC + x]$. How do I interpret it? I assume that if lower bound of interval is higher than 0.5 then I can conclude that my model is better than random one. What confuses me is that $AUC$ is in the middle of interval so it will always be inside CI. I think that maybe if my model was applied to some different observation, I would be 95% sure that its $AUC$ fit into CI. Am I right?
Thanks.
Best Answer
A confidence interval is an interval-estimate for some true value of a parameter. Let us (as an example) start with e.g. a confidence interval for the mean of a normal distribution and then move on to ROC and AUC so that one sees the analogy.
Assume that you have a random normal variable $X \sim N(\mu;\sigma)$. Where $\mu$ is the unknown population mean and, to keep it simple, let us assume that $\sigma$ is known.
We now draw a sample of size $n$ from the distribution of X, i.e. we get a sample $x_1, x_2, \dots x_n$. The goal is to have an idea about the unknown $\mu$ using the sample drawn. It is well known that the arithemetic average $\bar{x}=\frac{1}{n}\sum_i x_i$ is an unbiased (point) estimator for (the unknown) $\mu$ and that $[\bar{x}-1.96\frac{\sigma}{\sqrt{n}};\bar{x}+1.96\frac{\sigma}{\sqrt{n}}]$ is a $95\%$ confidence interval for (the unknown) $\mu$.
If we draw another sample $y_1, \dots , y_n$ from the distribtion of $X$ then, in the same way we will find another confidence interval for the (unknown) $\mu$ as $[\bar{y}-1.96\frac{\sigma}{\sqrt{n}};\bar{y}+1.96\frac{\sigma}{\sqrt{n}}]$.
So each time we draw a sample of size $n$ from the distribution of $X$, we find a confidence interval for the (unknown) $\mu$ and all these intervals will be different. The fact that it is a $95\%$ confidence interval means that, if we draw an 'infinite' number of samples of size $n$ from the distribution of $X$, and for each of these samples we compute the $95\%$ confidence interval, then $95\%$ of all these intervals (one interval for each sample) will contain the unknown $\mu$. (so sometimes , namely $5\%$ of the intervals, such an interval will not contain the unknown $\mu$, so sometimes you have bad luck.)
The same holds for the AUC, when you compute the AUC, you compute it from a sample, in other words what you compute is an estimate for the true unknown AUC. Similarly you can, for the sample that you have, compute a confidence interval for the true but unknown AUC. If you were able to draw an infinite number of samples, and for each sample obtained compute the confidence interval for the true AUC, then $95\%$ of these computed intervals would contain the true but unknown AUC.
Note that the interval is random, because it is computed from a random sample. The true AUC is not random, it is some unknown property of your population.
Unfortunately you can not draw an infinite number of samples, most of the time you have only one sample, so you will have to do it with one interval, but you are rather confident ($95\%$ of the so computed intervals will contain the true unknown AUC) that this interval will contain the true AUC. And yes, if the lower border of the interval is higer than 0.5 then you can be rather confident that your model is not the random model, but, as above, you may also have had bad luck with the sample.