Solved – McNemars Test Sample size calculation

mcnemar-testsample-sizestatistical significance

I've searched for this particular question but i can't seem to find the right answer.

Lets say i expect prob p01 = 0.2 and prob p10 = 0.3. I want a power of 0.8 and alpha of 0.05.

This website:

http://powerandsamplesize.com/Calculators/Compare-Paired-Proportions/McNemar-Z-test-2-Sided-Equality

gives that a sample size of 390 is needed for this calculation.

However, when I create a matrix in R with these properties:

     Yes   No

Yes 156 | 117

no 78 | 39

mcnemar.test in R gives a p value of 0.005 while I expected a value of 0.05.

Is this because the calculator on this website thinks in pairs?

If I create a matrix with the same properties but divided by 2 i do get the p value of 0.05:

   Yes   No

Yes 78 | 58.5

no 39 | 19.5

Is it safe to asume that the sample size calculated on this website (and also by the functions in R) should be divided by 2 ?

Best Answer

I have a guess:

Online calculator, you linked to, uses McNemar test in the context of two groups (Group A and Group B, they call them). Each entity in Group A has it's counterpart in Group B.

Common use of McNemar test is however, repeated measures case: we usually have one group and two measurments (Before and After, mostly) for each entity.

Notice that in first situation, sample size is number of entities in Group A plus number of entities in Group B, which is equivalent to number of entities in Group A times 2. But in contigency table, each entry is number of pairs (eg. number in Success-Success cell is number of pairs in which both entities succeded). So, total sum in contigency table is number of pairs, which is sample size divided by two.

In second situation, sample size is simply number of entities. Now, in contigency table, each entry is number of entities (eg. number in Success-Success cell is number of entities who had success Before as well as After). So, total sum in contigency table is now number of entities, which is sample size.

To sum up: online calculator uses (I think) first, and you use second setting. These two settings differ in definition what a sample size is.

And once again: it's a guess (to long to post it as a comment). I'm not sure if I'm right. Feedback apreciated.

Related Solutions

Solved – Sample size calculation Wilcoxon rank-sum test

I usually turn to simulation for power calculations for the Wilcoxon sign-rank test. I have my own function I use for this. Use it at your own risk, as I don't know that anyone has ever validated it.

You can read the function directly using

source("https://raw.githubusercontent.com/nutterb/StudyPlanning/master/R/sim_wilcoxon.R")

or install the package (I haven't been actively developing it for a couple years) using:

devtools::install_github("nutterb/StudyPlanning")

In order to make it work, you'll need to estimate distributions from each of the groups in your sample. In the example below, I've assumed one group follows a Poisson distribution with a mean of 2.1, and the other follows a Poisson distribution with a mean of 3.53. I've also assumed equal sample sizes. This yields an estimate power of 0.444.

set.seed(123)

sim_wilcoxon(n=22,                 # total sample size
            weights=list(c(1, 1)), # equal sample size per group
            rpois(lambda=2.1),     # distribution of first sample
            rpois(lambda=3.53),    # distribution of second sample
            nsim=1000)

  n_total n1 n2   k alpha power nsim pop1_param  pop2_param pop1_dist pop2_dist
1      22 11 11 0.5  0.05 0.444 1000 lambda=2.1 lambda=3.53     rpois     rpois

Sample Size Calculation – Formula for One Sample t-test

From the fragmentary and undocumented R code you show, I suppose you want to do a two-sided, one-sample t test at level $\alpha = 0.05$ based on a sample from a normal population with standard deviation $\sigma=1.91$ and hope for power $0.80$ to detect a difference in population means of $1.$

Several methods are in common use, and they may give slightly different answers.

Find sample size necessary to get power 80% using a comparable z-test. When the required $n$ is 30 or larger, the result will be approximately correct.
Use an exact formula for the power of such a t test, based on a non-central t distribution. Many intermediate level applied statistics texts and mathematical statistics texts show the formula, and software such as R will do the necessary computation for the noncentral t distribution.
Many statistical computer programs have 'power and sample' size procedures; most use the noncentral t distribution.
Simulation of many t tests for normal data of a trial sample size $n$ from a population with appropriate $\mu$ and $\sigma$ to find the proportion that reject (approximate power).

You have already seen computer output from R. Below is output from a recent release of Minitab statistical software. It gives $n = 31$ as the desired sample size--in agreement with your result from R.

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus ≠ null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 1.91


            Sample  Target
Difference    Size   Power  Actual Power
         1      31     0.8      0.805289

Finally, here is a simulation in R, showing that (in appropriate circumstances) $n = 31$ gives power about 80%. [I use a 'for' loop because it seems to be more widely understood than more elegant structures in R. With $m = 10\,000$ iterations one can expect about two decimal places of accuracy.]

set.seed(314)
n = 31;  mu.0 = 100;  mu.a = 101;  sg = 1.91
m = 10000;  t.stat = numeric(m)
for(i in 1:m) {
 x = rnorm(n, mu.a, sg)
 t.stat[i] = ( mean(x) - mu.0 )/( sd(x)/sqrt(n) )
 }
c = qt(.975, n-1);  c    # critical value
[1] 2.042272
mean(abs(t.stat) >= c)   # aprx power
[1] 0.8037

Note: If discrepancies among the various formulas and computational methods you used are small, that may be due to rounding errors or approximations. If discrepancies are large, you need to verify you have correct formulas and are using correct syntax in programs.

Best Answer

Related Solutions

Solved – Sample size calculation Wilcoxon rank-sum test

Sample Size Calculation – Formula for One Sample t-test

Related Question