Multiple Comparisons – Meaning of Post Hoc Tests like Tukey HSD

multiple-comparisonspost-hocterminologytukey-hsd-test

My original thought was “post hoc” in a post hoc multiple comparison procedure means that the procedure is taken after an omnibus test (such as ANOVA F test) gives a significant result.

But after reading Howell's Statistical Methods for Psychology, “post hoc” in a post hoc multiple comparison procedure means the procedure is planned after data collection, and “prior” in a prior comparison means the procedure is planned before data collection.

So was my original thought wrong?

Is Tukey–Kramer method (Tukey's HSD) post hoc?

How about Scheffe method?


A table from Howell's book (here is link to Google Books, although not having the table):

enter image description here

Best Answer

As always, your question implicitly asks for some authoritative answer that might very well not exist. Scheffé's method and Tukey's HSD are usually called post-hoc tests, used for unplanned comparisons and conducted after an omnibus test but that's not a requirement for all such methods.

The main argument for a distinction between planned and unplanned tests is that if you always intended to conduct a limited number of tests (planned contrasts), you don't necessarily need to adjust the error level. If, on the other hand, you are just reporting/testing the differences that look big (post-hoc tests), you might be “capitalizing on chance” and you should adjust not only of the tests you conduct/report but for all possible pairwise comparisons/contrasts in your design.

One issue with all this is that it makes the evaluation of the evidence and the result of a study contingent on the intentions of the experimenter, a most counter-intuitive and undesirable state of affairs. This is sometimes held as an argument against null-hypothesis significance testing as used within psychology.