Solved – Do you need to adjust p-values after Tukey’s HSD? How to do this correctly

multiple-comparisonstukey-hsd-test

I have multiple sets of comparisons I am running together. Normally I would do an FDR correction on the p-values to control the family-wise error rate. However, the p-values from a Tukey HSD are already adjusted for multiple comparisons (within a single test).

Does this pose problems for adjusting the p-values again?

As an example, let's say I am testing the difference among 3 treatments, first in group A (one test), and separately in group B (the second test). This gives me a total of 6 pairwise comparisons, 3 in each group. Normally I would adjust the resulting 6 p-values. But with Tukey HSD, the p-values have already been adjusted within each group (accounting for 3 comparisons), so to correct all 6 p-values as if they have not already been adjusted seems not quite right.

*EDIT: I neglected to say, this is after an omnibus ANOVA as well as simple effects test (because of significant interactions), both with corrected p-values. Having partitioned within levels of one factor, I proceed to Tukey's to examine combinations of the remaining factor.

Best Answer

Things I would suggest:

  1. Before running any a posteriori test such as Tukey's HSD, compute a p-value using an omnibus test, which will tell you if there are significant differences among any of the pairwise comparisons.
  2. IF the omnibus test indicates evidence that significant differences exist (e.g., p < 0.05), ensure that your data meet a criterion for homogeneity of variance (e.g., Levene's test).
  3. If the data meet your standard for homogeneity (e.g., p > 0.05), then use Tukey's HSD test to determine which parwise comparisons are specific and report an unadjusted p-value.

With this said, I think it is far more reasonable (and maybe preferable considering the general opinion of p-values, but if your boss/advisor/colleague/etc. wants p-values you may have to simply give them to him/her) is simply to report the omnibus p-value along with the group means and standard deviations. Any reader capable of interpreting the individual means should be able to tell which groups differ by a margin of any practical significance.

Related Question