Hypothesis Testing – Book Recommendation: Sample Size Determination for Hypothesis Testing of the Mean

hypothesis testingreferencessample-size

After a previous attempt to get answers here to what I found out was a nonsensical question I am looking for some good resources I can use to begin to understand the area.

The majority of Psychology experiment books in our library focus on statistics, data analysis, qualitative research methodologies and experiment design. But methods and techniques for determining sample size, with practical examples, seem to be few and far between, especially at a beginner level.

I have found a few books that I believe would be useful online but they are quite expensive and I would rather get feedback that they were worth the money and that there was nothing better out there.

Sample Size Determination and Power – Thomas P. Ryan
Sample Size Calculations: Practical Methods for Engineers and
Scientists. Paul Mathews

So can anyone recommend books or other resources in the area, specifically for determining the sample size for hypothesis testing of the mean?

A brief overview of my research area:
I am undertaking research to determine which particular features of a website or software interface affect a users perception using metrics such as trust, fairness, expertise etc. Each participant will rate 9 separate designs, each with a different distortion which I will then compare to the control. Participants will be crowdsourced online. Hope this helps.

Best Answer

Some very basic sample size calculations are discussed here:

http://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

http://www.itl.nist.gov/div898/handbook/prc/section2/prc242.htm

For a basic introduction, the book by Moore and McCabe - Introduction to the Practice of Statistics covers some of the basics in chapter 6.

Some deeper discussion is in

Russell V.Lenth, 2001.
"Some Practical Guidelines for Effective Sample Size Determination"
The American Statistician, August 2001, Vol. 55, No.3

Edit:

I just noticed that I had left out Jacob Cohen's book. I meant to mention it - Statistical Power Analysis for the Behavioural Sciences

I don't know this book, but I've seen some people recommend

Chow S, Shao J, Wang H. 2008.
Sample Size Calculations in Clinical Research.
2nd Ed. Chapman & Hall/CRC Biostatistics Series.

As I said in comments, I tend to use simulation.

Related Solutions

A/B Testing – How to Safely Determine Sample Size for A/B Testing

The most common method for doing this kind of testing is with binomial proportion confidence intervals (see http://bit.ly/fa2K7B)

You won't be able to ever know the "true" conversion rate of the two paths, but this will give you the ability to say something to the effect "With 99% confidence, A is more effective at converting than B".

For example: Lets assume that you have run 1000 trials down path A. Of these 1000 trials, 121 were successful conversions (conversion rate of 0.121) and we would like a 99% confidence interval around this 0.121 result. The z-score for 99% confidence intervals is 2.576 (you just look this up in a table), so according to the formula: $$ \begin{aligned} \hat p &\pm 2.576\left(\sqrt{\frac{0.121 * (1 - 0.121)}{1000}}\right) \\ \hat p &\pm 0.027 \end{aligned} $$ So with 99% confidence we can say that $0.094 \le \hat p \le 0.148$, where $\hat p$ is the "true" conversion rate of process A.

If we construct a similar interval for process B, we can compare the intervals. If the intervals don't overlap, then we can say with 98% confidence that one is better than the other. (Remember, we're only 99% confident about each interval, so our overall confidence about the comparison is 0.99 * 0.99)

If the intervals do overlap, then we have to run more trials, or decide that they are too similar in performance to distinguish, which brings us the tricky part - determining $N$, the number of trials. I'm not familiar with other methods, but with this method, you aren't going to be able to determine $N$ up front unless you have an accurate estimate of the performance of both A and B up front. Otherwise, you are just going to have to run trials until you get samples so that the intervals separate.

Best of luck to you. (I'm rooting for process B, by the way).

Solved – Sample Size Determination for Rare Event

I hope this is helpful, but I'm not very knowledgeable myself.

I think a power analysis would make sense here. You enter all the variables except for one, which is calculated. In your case enter everything except for n, and leave that null. the results will tell you how many people, per group, you need to meet your requirements.

If you're using R, the code is

power.prop.test(n = NULL, p1 = NULL, p2 = NULL, sig.level = 0.05,
                power = NULL,
                alternative = c("two.sided", "one.sided"),
                strict = FALSE)

and according to the help page

n   
Number of observations (per group)

p1  
probability in one group

p2  
probability in other group

sig.level   
Significance level (Type I error probability)

power   
Power of test (1 minus Type II error probability)

alternative 
One- or two-sided test

strict  
Use strict interpretation in two-sided case

I apologize if I've made an error myself! But hopefully this is better than nothing.

Best Answer

Related Solutions

A/B Testing – How to Safely Determine Sample Size for A/B Testing

Solved – Sample Size Determination for Rare Event

Related Question