Solved – t-Test with small sample sizes (10)

hypothesis testingstatistical significancet-testwilcoxon-mann-whitney-test

I am looking to compare the means of two sample populations of size 10. Ideally I would like to use an independent t-Test however I am concerned that the sample size is too small, as I don't know for sure that they are normally distributed.

The samples are final scores in a video game which was played between subjects under two different conditions (one independent variable).

I can't seem to find a clear answer on what test I should use to make this simple comparison – any advice would be really appreciated.

Best Answer

It turns out that someone else on StackExchange asked about t-tests and sample sizes, and the summary appears to be that yes, the t-test is valid even in small sample sizes. You merely need to approximately satisfy the t-test's assumptions. So, you could just go ahead and do a t-test. Your power is what it is with a sample of 10 in each arm.

I've heard one professor say that permutation tests are appropriate for inference in small samples. I haven't yet found anyone saying that they are better than t-tests in small samples, but they are basically only practical in smaller samples, and the link below says they can be a good check of a t-test result. In a permutation test, you would (by my reading of the link) assume that the between-group difference is actually zero, then using your data, calculate all possible permutations of the between-group difference. You're basically calculating an exact p-value for the difference - remember that p-values mean that assuming the null hypothesis is true (i.e. that the difference between groups is zero), what's the probability that you'd have seen the between-group difference you got in your sample?

Related Question