Power Analysis – Adjusting Calculations with Finite Population Correction (FPC)

finite-populationhypothesis testingsample-sizesamplingstatistical-power

Is it correct (perhaps at least as a good approximation) to determine sample size via a power analysis that assumes an infinite population and then to adjust the required sample size by a finite population correction?

For example, I am using the formula from Fleiss:

enter image description here

To calculate the sample size required to test the difference between two independent proportions, with unequal sample sizes.

If I get out of this formula that the two samples sizes are:

n1= 300

n2= 600

(as r in the formula, the ratio n2/n1, was set to 2) can I then apply the FPC to the total sample size to correct for the finite population?

enter image description here

For example, say the population size is really 1000. Then, can I apply the correction to the total sample size (300+600=900) and recalculate the sample sizes from the two populations as:

new_total_sample = $\frac{900*1000}{900+(1000-1)}=474$ and then allocate between the two populations as new_n1 = 158 and new_n2 = 316?

Thanks!

ADD After Dr. Lumley's response:

I. I am a little confused on the difference between a hypothesis test and a confidence interval for a finite population difference. Are you saying it is legitimate to estimate the confidence interval of say, the difference between the proportion of men and women answering 'Yes' but not a p-value to determine if it is zero (i.e. to reject the null)?

In the language of your package, I am thinking of

design  <- svydesign(id=~1, fpc=~N,data=experimentData)
svyglm(respondedYES~as.factor(isMale), design=design, family=binomial)

where you would construct a CI around the coefficient for isMale

II. Also, I understand I think, the issue with the example of how the proportion of men and women could never be the same given your lack of a common factor argument, but what about hypothesis testing with a finite population in the case of an experimental design (where the researcher controls the treatment assignment) – where this issue is not present? That is the realm I am trying to learn how (if) the fpc can be used.

Best Answer

Update: 2014-02-06: changed text to be more emphatic that fpc should not be used in a causal analysis **Update: 2014-02-04: impact of the randomized experimental design

This question has raised some fundamental issues.

You stated in your update that a researcher can control the make-up of the experimental groups. Not so. Even if one randomized an entire population, there would be imbalance, perhaps trivial, in every variable. Even with some kind of balancing algorithm, which would destroy the randomization, one can never arrange for identity of the means of the outcome variable, yet unmeasured.

You also asked Tom Lumley:

Are you saying it is legitimate to estimate the confidence interval of say, the difference between the proportion of men and women answering 'Yes' but not a p-value to determine if it is zero (i.e. to reject the null)?

I think that's what Tom meant, and I agree with its application to descriptive statistics; ~~I'm not sure that it applies~~ It does not apply to causal analyses, including those generated by an experiment. Your particular example is a borderline case, as you intend the results to apply to a single population at a particular time. If someone asked you to project your findings to another setting or to another time period, the confidence interval calculation ~~probably~~ should not include the fpc.

Some additional insight can be gained by considering the experimental design as part of the sample design. If the initial random sample is of size $n$, randomization produces two random sub-samples of size $n_1 = n/2$ and $n_2 = n/2$. (For the theory that follows, $n_1$ and $n_2$ need not be equal.) Let $\overline{y}_1$ and $\overline{y}_2$ be the means of the sub-samples; proportions are special cases. In this scenario, which conforms to the absence of a treatment effect, it can be shown (Cochran, 1977, problem 2.16, p. 48) that:

\begin{equation} Var(\overline{y}_1 -\overline{y}_2) = S^2\left(\frac{1}{n_1} +\frac{1}{n_2}\right) \end{equation}

where $S^2$ is the population variance and variation is over repetitions of the sampling and randomization. Notice: no fpc.

Update: one of the few established uses of hypothesis tests + FPCs for finite populations: lot quality assurance sampling (LQAS)

I agree with Tom's answer. Hypothesis testing rarely has a place in finite population questions, but confidence intervals certainly do. One good use of hypothesis tests per se in finite populations is lot quality assurance sampling (LQAS), which tests whether the rate of some event (e.g. vaccination) in a geographic area is too high or too low. Note that, unlike the question at hand, there is no hypothesis of zero difference. The null hypothesis is that the rate is < K, and the alternative that is it is $\geq$K. See, at Google Scholar.

Robertson, Susan E, Martha Anker, Alain J Roisin, Nejma Macklai, Kristina Engstrom, and F Marc LaForce. 1997. The Lot quality technique: a global review of applications in the assessment of health services and disease surveillance. Relation 50, no. 3/4: 199-209.

Lemeshow, Stanley, and Scott Taber. 1991. Lot quality assurance sampling: single-and double-sampling plans. World Health Stat Q 44, no. 3: 115-132.

Original Answer

Using the fpc to reduce sample size makes no sense unless intend you use it in the the hypothesis-testing statistic. But that would be an error: the fpc should not be used when testing hypotheses [added about "no difference"].

The reasoning is interesting (Cochran, 1977, p.39): It is seldom of scientific interest to ask if a null hypothesis (e.g. that two proportions are equal) is exactly true in a finite population . Except by a very rare chance, the null hypothesis will never be true, as one would discover by enumerating the entire population. Therefore hypothesis tests on samples from finite populations are done from a "super-population" viewpoint. See also Deming (1966) pp 247-261 "Distinction between enumerative and analystic studies"; Korn and Graubard (1999), p. 227.

References

Cochran, W. G. (1977). Sampling techniques (3rd ed.). New York: Wiley.

Deming, W. E. (1966). Some theory of sampling. New York: Dover Publications.

Korn, E. L., & Graubard, B. I. (1999). Analysis of health surveys (Wiley series in probability and statistics). New York: Wiley.

Related Solutions

Finite Population Correction Factor – Detailed Explanation and Use

The threshold is chosen such that it ensures convergence of the hypergeometric distribution ($\sqrt{\frac{N-n}{N-1}}$ is its SD), instead of a binomial distribution (for sampling with replacement), to a normal distribution (this is the Central Limit Theorem, see e.g., The Normal Curve, the Central Limit Theorem, and Markov's and Chebychev's Inequalities for Random Variables). In other words, when $n/N\leq 0.05$ (i.e., $n$ is not 'too large' compared to $N$), the FPC can safely be ignored; it is easy to see how the correction factor evolves with varying $n$ for a fixed $N$: with $N=10,000$, we have $\text{FPC}=.9995$ when $n=10$ while $\text{FPC}=.3162$ when $n=9,000$. When $N\to\infty$, the FPC approaches 1 and we are close to the situation of sampling with replacement (i.e., like with an infinite population).

To understand this results, a good starting point is to read some online tutorials on sampling theory where sampling is done without replacement (simple random sampling). This online tutorial on Nonparametric statistics has an illustration on computing the expectation and variance for a total.

You will notice that some authors use $N$ instead of $N-1$ in the denominator of the FPC; in fact, it depends on whether you work with the sample or population statistic: for the variance, it will be $N$ instead of $N-1$ if you are interested in $S^2$ rather than $\sigma^2$.

As for online references, I can suggest you

Solved – R survey package: finite population correction affects point estimate in addition to the variance estimate

yes, they will give different estimates. ?svydesign says "If population sizes are specified but not sampling probabilities or weights, the sampling probabilities will be computed from the population sizes assuming simple random sampling within strata."

looking inside survey:::svydesign.default

if (is.null(probs) && is.null(weights)) {
    if (is.null(fpc$popsize)) {
        if (missing(probs) && missing(weights)) 
            warning("No weights or probabilities supplied, assuming equal probability")
        probs <- rep(1, nrow(ids))
    }
    else {
        probs <- 1/weights(fpc, final = FALSE)
    }
}

so if weights are not specified by the user but the fpc is, then the stratified fpc gets used in the computation for the weights (which will affect point estimates as well as variance calculations)

library(survey)
data(api)

dstrat1<-svydesign(id=~1,strata=~stype, data=apistrat, fpc=~fpc)
dstrat2<-svydesign(id=~1,strata=~stype, data=apistrat)

svymean( ~ api00 , dstrat1 )
svymean( ~ api00 , dstrat2 )

Best Answer

Related Solutions

Finite Population Correction Factor – Detailed Explanation and Use

Solved – R survey package: finite population correction affects point estimate in addition to the variance estimate

Related Question