We have recently conducted a cluster analysis for an open card sorting task in which 19 experts grouped 112 items each. We achieved decent silhouette values for the cluster analysis but have been asked by reviewers to provide evidence that the number of experts we used was sufficient for this task (I think they are expecting something kind of power analysis).
We have used Kendall's Coefficient of Concordance to examine the concordance in individual sort patterns between participants (this came out as .25), but it seems they want something more. After looking around I cannot find anything which might suggest a suitable sample size or even that sample size is not important and I'm pretty sure power analyses don't apply for cluster analysis. Is there anything I can report to show my sample size is sufficient (or not if the case may be)? I am using R to analyse the data.
BTW by sample size I mean number of participants not number of items included in the cluster analysis.
Best Answer
It sounds like they want you to do a post hoc power analysis. The purpose of a power analysis is to optimize the study that will be conducted (e.g., to increase the likelihood of getting a statistically significant result). A power analysis is always speculative - you have to guess what the reality is. A power analysis is NOT data analysis. It sounds like you've already conducted the study. So the original power analysis cannot tell you ANYTHING about the results. A "new" power analysis could tell you how to optimize the NEXT study, but cannot inform the current study. All your results are represented by your statisitcal analysis (your parameter estimates, confidence intervals, and p-values). There have been many articles in the statistical literature on why post hoc power analysis makes no sense and can only lead to contradictions. Uninformed researchers often resort to this to try to explain why their results were NOT statistically significant - but this is not possible. If your results WERE statistically significant, well that's your results (you can't argue with statistical significance). Search for the paper "The Abuse of Power: the Pervasive Fallacy of Power Calculations as Data Analysis"- http://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf pu blished in The American Statistician. It's sad when reviewers aske researches to do ridiculous things that make no sense statistically.