I was reading an entry level statistics textbook. In the chapter on maximum likelihood estimation of the success proportion in data with binomial distribution, it gave a formula for calculating a confidence interval and then nonchalantly mentioned
Consider its actual coverage probability, that is, the probability that the method produces an interval that captures the true parameter value. This may be quite a bit less than the nominal value.
And goes on with a suggestion to construct an alternative "confidence interval", which presumably contains the actual coverage probability.
I was confronted with the idea of nominal and actual coverage probability for the first time. Making my way through old questions here, I think I got an understanding for it: there are two different concepts we call probability, the first being how probable it is that a not-yet-happened event will produce a given result, and the second is how probable is that an observing agent's guess for the result of an already-happened-event is true. It also seemed that confidence intervals only measure the first type of probability, and that something called "credible intervals" measure the second type of probability. I summarily assumed that confidence intervals are the ones which calculate "nominal coverage probability" and credible intervals are the ones which cover "actual coverage probability".
But maybe I have misinterpreted the book (it is not entirely clear on whether the different calculation methods it offers are for a confidence interval and a credible interval, or for two different types of confidence interval), or the other sources I used to come to my current understanding. Especially a comment which I got on another question,
Confidence intervals for frequentist, credible for Bayesian
made me doubt my conclusions, as the book did not describe a Bayesian method in that chapter.
So please clarify if my understanding is correct, or if I have made a logical error on the way.
Best Answer
In general, the actual coverage probability will never be equal to the nominal probability when you are working with a discrete distribution.
The confidence interval is defined as a function of the data. If you are working with the binomial distribution, there are only finitely many possible outcomes ($ n+1$ to be precise), so there are only finitely many possible confidence intervals. Since the parameter $ p $ is continuous, it's pretty easy to see that the coverage probability (which is a function of $ p $) can do no better than be approximately 95% (or whatever).
It is generally true that methods based on the CLT will have coverage probabilities below the nominal value, but other methods can actually be more conservative.