Solved – Adjustment of p-values for multiple-comparisons in lsmeans

lsmeansmultiple-comparisonsr

Using R's lsmeans, I'm testing the pairwise comparisons among factor1 levels for each different combination of factor2 and factor3 levels:

lsmeans(model,pairwise~factor1|factor2*factor3,adjust="holm")

(model was produced by lmer).

The last line in the output states:

P value adjustment: holm method for 3 tests

It seems that lsmeans corrects MCP only within each combination of factor2 and factor3, but not across the combinations (In my data there are 14 combinations of factor2 and factor3 and 3 levels within factor1, so the number of total tests is 42).

  1. Is this behavior reasonable? Doesn't it inflate the overall type I error? Why is this the default?
  2. How can one make lsmeans correct across combinations (i.e. for the total number of tests?)

Best Answer

You can exert more control over what you get if you do the contrasts in a separate step:

lsm = lsmeans(model, ~ factor1 | factor2*factor3)
comps = pairs(lsm)
summary(comps, by = NULL, adjust = "holm")

The by = NULL part of the last statement tells it to ignore the by grouping and thus treat it as one family of tests.

To answer part of the first question, yes, it does inflate the overall type I error, if by that you mean the whole family of 42 tests. On the other hand, I could ask the same question about the results of your one-shot command that includes the least-squares means (I think there are $3\times42=126$ of them) and the 42 pairwise comparisons you requested, for a total of 168 tests: Is it perhaps necessary to protect the overall type I error for that set of 168 tests?

There is always a choice of what family of tests you are applying a multiplicity adjustment to. I am not sure if the per-group adjustment is the most reasonable; I think it is what most users expect, and that is why it is the default. At any rate, one of the design decisions was to try to state explicitly how the adjustments are made, so people don't make the wrong assumption about what they see.

Any comments of whether that is the right default?

Related Question