Multivariate Analysis – Comprehensive Guide to Post Hoc Power Analysis in Statistical Studies

multivariate analysisstatistical-power

I have an experiment with 3 equal group sizes and 4 measures. I think the simple null hypothesis is that the three groups will be the same. Most people, however, believe that group A should do best in measure 1, group B should do best in measure 2, group 3 should do best in measure C, and they should all be the same in measure 4, both when comparing within and between groups. It is quite possible my group size is too small to come to any conclusions. I base this on the tried and true approach that my N is smaller than their N.

I started off by doing a multivariate GLM (using SPSS) and found that the effect group is not significant (F = 1.8, p = 0.1). Looking at the data suggests that even if there is a difference between the groups, that it doesn't likely follow what is expected (e.g., group A does not do the best in measure 1 and is best in measure 2). My thinking is that maybe a post hoc power analysis might tell me something useful. SPSS spits out an observed power of 0.69. At this point I am lost. Does the observed power tell me anything useful? Am I going about this all wrong?

Best Answer

Post hoc power analyses are at best useless and are often misleading, read "The Abuse of Power" (Hoenig and Heisey, American statistician, vol 55, issue 1, 2001) for more details.

What might be more useful is confidence intervals on your measures, they can tell you if your original ideas could still be meaningful and if the plausible differences are large enough to care about. These will be much more meaningful than a transformation of the p-value (which is all that the post hoc power is) that is usually interpreted wrongly.