Solved – Simultaneous tests for general linear hypothesis question

confidence intervalmultiple-comparisonspost-hocprobitr

I have run a probit regression and am now trying to run post-hoc tests. I am trying to compare differences between a 3 level factor variable.

I am confused about the difference between running a 'simultaneous tests for general linear hypotheses' and running the same thing but with a 'Tukey' adjustment- they get very similar answers but is either 'better' or 'worse'? Or does it not matter? For example:

library(lsmeans)
lsmeans(m1, pairwise~Name.Origin, adjust="tukey") 

library(multcomp)
summary(glht(m1, lsm(pairwise~Name.Origin)))

In addition, is there a more formal name for the first method which just fits the model?

Best Answer

If you specify adjust = "mvt" in the lsmeans call, you'll get exactly the same results as the glht call (except for minor differences due to the fact that the computations are simulation-based). The difference would come if you summarize the tests in the glht object with some option other than the one-step method (which is the default). The one-step method protects the error rate for simultaneous confidence intervals, which is stronger (and hence more conservative) than the step-down methods. The mvt method is the exact one-step method when the distributional assumptions hold.

Also, in a nicely balanced experiment with homogeneous errors, there is no difference between the Tukey method and the mvt method. That is, the Tukey method is the mvt method for the particular covariance structure encountered in such a balanced design.

Related Question