We performed an experiment (usability testing), which had two independent samples, one group with 5 participants and the other with 24. Based on the analysis of the distribution of the data (Shapiro-Wilk test), we concluded that we operate with data where some measurements distributed normally and others do not confirm to normal distribution.
I was wondering which test would be the most appropriate for data that is normally distributed and which for the data that is not normally distributed in our case, since we compare a group with 5 participants with a group that has 24?
We found T-Test (for normally distributed data) and Mann Whitney test (for data that is not normally distributed), however, we have two groups that are uneven and based on the conflicting information we are not sure if we can perform these tests.
Thank you for your answer!
Best Answer
A simple yet robust approach could be to compute the difference between medians and then compute 95% confidence interval of such statistic with bootstrap (percentile). Such confidence interval would be adequate for inference.
You can find here some useful references:
http://r.789695.n4.nabble.com/CI-for-the-median-difference-td4399508.html
http://r.789695.n4.nabble.com/CI-for-the-median-difference-td4399508.html
http://www.statalist.org/forums/forum/general-stata-discussion/general/564770-hypothesis-testing-for-bootstrapped-differences-in-medians-in-a-randomized-clinical-trial
Yet, the small samples (5 cases in one group), limits substantially external validity, even if your inferential estimates were quite precise (anyway very unlikely).