I would like to calculate a confidence interval for the probability of a process succeeding after a given amount of time by running the process $n$ independent times and recording the time at which each run succeeded. Essentially, I would like to add confidence intervals to a plot like this:
I think this can be modelled as the probability of $n$ coins having ever come up heads after $t=1,2,\ldots$ flips. If I had $n$ independent runs of each different number of flips, I would expect to model the confidence interval at each number of flips as a binomial proportion confidence interval, e.g.,
- flip all coins ($n_1 = n$) once each (i.e., $t=1$) and count each coin that came up heads ($s_1$),
- flip all coins ($n_2 = n$) twice each (i.e., $t=2$) and count each coin that came up heads on either flip ($s_2$),
- etc.
and then calculate the Clopper–Pearson interval for each $s_t/n$.
But in this scenario the subsequent flips aren't independent and it is equivalent to,
- flipping all coins ($n_1 = n$) once (i.e., $t=1$) and counting the number of heads ($s_1$),
- flipping only the coins that weren't heads ($n_2 = n-s_1$) again (i.e., $t=2$) and counting the cumulative number of heads ($s_1 + s_2$),
- etc.
In this cumulative scenario, I could still calculate the Clopper-Pearson interval at each $t$ from $\frac{\sum_{i=1}^t s_i}{n}$, but is that correct? It seems like it ignores the time relationship…
I'm sure this is a common problem and I am just thinking about it wrong or not using the right terms/language, so any direction is appreciated! If it is relevant, I am actually interested in the problem when $t$ is a real valued number (e.g., the amount of time a process has been running).
Best Answer
This type of time-to-event data calls for survival analysis. The survival function for a distribution is just 1 minus its cumulative distribution function.
Although survival analysis typically is used for failure-type events (like death) there's no reason why you can't treat your "successes" as events. Standard statistical software can typically handle survival data, including "censored" event times at which you might have decided to stop observing for an event because you were waiting too long.
For a simple description of the characteristics of the process as you show, this is basically what is done with a Kaplan-Meier survival curve, for which there are several choices of confidence interval (CI). If you want to plot the cumulative distribution instead of the survival curve, just subtract all survival values (point estimates and CI limits) from 1. If you have different machines used in the evaluation, with each machine evaluated multiple times, the within-machine correlations can be accounted for in the CI estimates.