proportion;small-samplestandard error
Standard error for a proportion, when n > 5 and np > 5 is calculated as $$SE = \sqrt{ \frac{p(1-p)}{n}}$$ where p is the proportion and n is sample size.
However, for even smaller samples, we were given an equation $$SE=\frac{1-e^{\frac{log(0.05)}{n}}}{1.96}$$
Is the second formula a valid way to calculate standard error of a proportion for very small samples? What is the rationale behind it?
To be perfectly clear what we're discussing, let's establish some concepts and terminology. A nice model for proportions is the binary urn: it contains balls colored either silver ("success") or fuchsia ("failure"). The proportion of silver balls in the urn is $p$ (but this is not the "proportion" we will be talking about).
This urn provides a way to model a Bernoulli Trial. To obtain one realization, thoroughly mix the balls and blindly draw one out, observing its color. To obtain additional realizations, first reconstitute the box by returning the drawn ball, then repeat the procedure a predetermined number of times. The sequence of $n$ realizations can be summarized by the count of its successes, $X$. It is a random variable whose properties are completely determined by $n$ and $p$. The distribution of $X$ is called a Binomial$(n,p)$ distribution. The (experimental, or "sample") proportion is the ratio $X/n$.
These figures are barplots of probability distributions for various binomial proportions $X/n$. Most noteworthy is a consistent pattern, regardless of $n$, in which the distributions become narrower (and the bars correspondingly higher) as $p$ moves from $1/2$ on down.
The standard deviation of $X/n$ is the standard error of proportion mentioned in the question. For any given $n$, this quantity can depend only on $p$. Let's call it $\operatorname{se}(p)$. By switching the roles of the balls--call the silver ones "failures" and the fuchsia ones "successes"--it's easy to see that $\operatorname{se}(p) = \operatorname{se}(1-p)$. Thus the situation where $p=1-p$--that is, $p=1/2$--must be special. The question concerns how $\operatorname{se}(p)$ varies as $p$ moves away from $1/2$ towards a more extreme value, such as $0$.
Because everyone has been shown figures like these early in their education, everybody "knows" the widths of the plots--which are measured by $\operatorname{se}(p)$--must decrease as $p$ moves away from $1/2$. But that knowledge is really just experience, whereas the question seeks a deeper understanding. Such understanding is available from a careful analysis of Binomial distributions, such as Abraham de Moivre undertook some 300 years ago. (They were akin in spirit to those I presented in a discussion of the Central Limit Theorem.) I think, though, that some relatively simple considerations might suffice to make the point that the widths must be widest near $p=1/2$.
It is clear that we should expect the proportion of successes in the experiment to be close to $p$. The standard error concerns how far from that expectation we might reasonably suppose the actual outcome $X/n$ will lie. Supposing, without any loss of generality, that $p$ is between $0$ and $1/2$, what would it take to increase $X/n$ from $p$? Typically, around $pn$ of the balls drawn in an experiment were silver and (therefore) around $(1-p)n$ were fuchsia. To get more silver balls, some of those $p n$ fuchsia outcomes had to have differed. How likely is it that chance could operate in this way? The obvious answer is that when $p$ is small, it's never very likely that we're going to draw a silver ball. Thus, our chances of drawing silver balls instead of fuchsia ones are always low. We might reasonably hope that by pure luck, a proportion $p$ of the fuchsia outcomes could have differed, but it seems unlikely that many more than that would have changed. Thus, it is plausible that $X$ would not vary by much more than $p\times (1-p)n$. Equivalently, $X/n$ would not vary by much more than $p(1-p)n/n = p(1-p)$.
Thus the magic combination $p(1-p)$ appears. This virtually settles the question: obviously this quantity peaks at $p=1/2$ and decreases to zero at $p=0$ or $p=1$. It provides an intuitive yet quantitative justification for assertions that "one extreme is more limiting than the other" or other such efforts to describe what we know.
However, $p(1-p)$ is not quite the correct value: it merely points the way, telling us what quantity ought to matter for estimating the spread of $X$. We have ignored the fact that luck also tends to act against us: just as some of the fuchsia balls could have been silver, some of the silver balls could have been fuchsia. Accounting for all the possibilities rigorously can get complicated, but the upshot is that instead of using $p(1-p)n$ as a reasonable limit on how much $X$ could deviate from its expectation $pn$, to account for all the possible outcomes properly we have to take the square root $\sqrt{p(1-p)n}$. (For a more careful account of why, please visit (https://stats.stackexchange.com/a/3904.) Dividing by $n$, we learn that random variations of the proportion $X/n$ itself should be on the order of $\sqrt{p(1-p)n}/n = \sqrt{\frac{p(1-p)}{n}},$ which is the standard error of $X/n$.
Best Answer
No.
The standard error of the mean (SEM or standard error) is a value representing how close the 'true' population mean is likely to be to the sample mean. It is related to the standard deviation (SD) - it is the SD divided by the square root of sample size - and it can be used along with the sample mean to derive a 95% confidence interval for the population mean (the 95% confidence interval for the population mean is the sample mean +/- 1.96*SEM).
For 'large' samples (two rules of thumb in common use are when np and n(1-p) are both greater than 5 or when they are both greater than 10) we can use a normal approximation. However, for small samples or extreme proportions, where the rule of thumb is not true, the interval within which the population mean is likely to be found is not symmetrical about the sample proportion. This means that the SD and the SEM cannot be usefully defined. The best alternative is to calculate and report a confidence interval for the population mean; this page includes a calculator for this using the Clopper-Pearson method.
See also the 2001 paper by Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion". Statistical Science. 16 (2): 101–133 which is a fairly accessible overview of the problem and some approaches to fixing it.
(It is worth noting that the above paper actually recommends either the Wilson interval, the Agresti interval or the Jeffries interval, but I couldn't find an online calculator for those; there are many in software packages. For example, in Matlab
[p, ci] = binofit([positives], [sample size]) gives you a confidence interval via the Agresti method).