I have performed a univariate Cox proportional hazards regression for an admittedly miniscule group of 25 samples, and consequently received a certain p-value (significant) and an associated confidence interval for the hazard ratio. The goal is not to create an applicable predictor or classifier, but rather to simply find out whether the variable of interest could be associated with the outcome. Now I need to decide whether or not this sample size is sufficient to trust the test, because a reviewer has asked about it. How do I convince either myself that the test is unreliable at this sample size, or, alternatively, the reviewer that the sample size is sufficient to trust the p-value and the confidence interval? I have looked into post hoc power analysis, but find that all methods seem to be designed for two group comparisons. There are also plenty of opinions that post hoc tests are meaningless to begin with.
I am not looking for an opinion about whether or not the size is sufficient, but rather for methods or references that would allow one to determine a minimum sample size required to trust the analysis. There is a significant riskt that the question is misguided in some way, due to a lack of knownledge, but it would good to know why it is misguided in that case.
Best Answer
Not an opinion, it is demonstrable fact. A post hoc power analysis doesn't tell you anything the p-value doesn't tell you. It is considered a huge faux pas to include on of these in your analysis, and your reviewer should be made aware of that in a polite and respectful way. See my previous answer here.
There must be some reason why you proceeded with the analysis. I think at this stage, being honest with the reviewer is all you can do. Trying to back pedal and justify the same size with existing studies might help, but I think it would be very dishonest.