I'm having trouble figuring out the appropriate way to place a confidence interval on a ratio I'm estimating. It's for a social science application, but I'll present the problem via coin-flips as I think its clearer.
Imagine you have $N=100$ coins, and you are going to flip each coin 3 times. Your null hypothesis H0 is that all the coins are fair, $p=q=.5$. But you suspect (H1) that actually some of the coins are fake such that they always return heads (which we will set equal to 1). You flip each coin three times, such that your results are N binomial variates of size=3.
I can observe the proportion of 'all heads' trials out of the N coins, and I can also calculate the predicted number of 'all heads' trials under the null hypothesis, which is $p^3N$. Say I observe 16 'all heads' trials. Under the null I expected to observe $(.5\times.5\times.5)N=12.5$ 'all heads' trials – how can I tell if these numbers are meaningfully different?
I know that the variance is $Npq$ for the proportion of successes $n$ in $N$ coin flips, but that doesn't seem the right way to go. In particular, the expected value of my result is $p^3N$, should I be treating an 'all heads' outcome like a success in a bernoulli trial of size $N$, and thus calculate variance as $Np^3(1-p^3)$?
Best Answer
Off the top of my head:
I see it as a binomial where the experiment is "throwing a coin 3 times", with N=100, just like you suggested.
Hence the number of "3 heads" $X$ follows a binomial distribution. $$ X \sim B(n=100,\; p=1/8) $$
This can be approximated by a normal law $$ N(\mu=pn,\; \sigma² = p(1-p)n) $$
So if you want say a 95% confidence interval (z=1.645), you will reject H0 if $$ X > \mu + z \sigma = 17.94 \approx 18$$
So if when you get 18 or more "3 heads" you conclude "something is fishy!!", you'll be wrong only about 5% of the time.