A health insurance company here presents rating lists of hospitals based of their SMR (standardized mortality/morbidity ratio) values (observed/expected, i.e. the higher the worse). They use the confidence intervals of SMR as well, which are wide in a small number of cases and narrow in large samples. So far, so good. It follows: they consider the best hospitals as those with SMR below the 20 percentile with lowest values of the upper ends of their confidence intervals and the worst hospitals as those with SMR above the 80 percentile with highest values of the lower ends of their confidence intervals – regardless whether the whole confidence interval of a particular hospital includes 1 or not. As we asked about a rationale for this they argued that the higher value of the lower end of the confidence interval shows the higher probability of the undesirable outcome.
They even give an example. Two hospitals perform the same surgery with 96% success: the hospital A on 500 patients and the hospital B on 50 patients. They mean that the achieving the success value of 96% success is statistically more likely in the hospital A, because their confidence interval is 94 – 98% there. The hospital B would have the confidence interval of 84 to 100% – thus, not that nice.
I always thought that the confidence intervals just show the range where the true value lands with 95% chance (well, for 95% intervals) and that they should be always observed as a whole, not just lower end or upper end. As I read somewhere, confidence interval shows where the bomb lands with the 95% probability, not where the 95% of all the bombs land. For the latter there are tolerance intervals that are calculated differently. In example above there would be no difference between two hospitals on the basis of the mentioned confidence intervals. And it a confidence interval of a SMR includes 1 you cannot claim the result is better or worse as the expected value – regardless how high or low one end of this confidence interval is located.
I’m surprised however since it is a large company with the own statistic department. Please give me an advice whether I interpet the things correctly or miss something.
Thank you!
Best Answer
Your understanding of confidence intervals is a little flawed - they are not where the true value lands 95% of the time - its where, if we repeatedly resampled the population and conducted the same study, 95% of the estimates would fall.
There's no reason the lower or upper bound of a confidence interval couldn't be considered separately for some questions - you implicitly do so in this part of your question: "regardless whether the whole confidence interval of a particular hospital includes 1 or not", which in essence is the same as "Is the lower bound below 1 for estimates above one?" and "Is the upper bound above one for estimates below 1". That kind of ad hoc significance test is invariably only considering one of the two bounds.
As for the meat of your question, there are definitely some issues with using the confidence intervals the way they are, and I'm a little skeptical of their approach. It inherently penalizes small hospitals, as they have wider confidence intervals by virtue of their size and the number of events that occur. This may be an overcorrection for a number of rating schemes that penalize the opposite direction, favoring small hospitals (Consumer Reports, I'm looking at you). More generally, I'm skeptical about rating hospitals against each other generally.