I need assistance with interpreting the outcome of my pairwise comparisons from my datasets. I've been running a glmer mixed models and selecting the best model using the AIC criteria. This is the ANOVA outcome from my model which had the lowest AIC:
Anova(lm13, type="III", test.statistic = "Chisq")
Analysis of Deviance Table (Type III Wald chisquare tests)
Response: Caspase3
Chisq Df Pr(>Chisq)
(Intercept) 9.6287 1 0.0019156 **
Post 1.8869 1 0.1695546
Adult 2.0416 1 0.1530531
Pre 7.4014 1 0.0065174 **
Age 1.4426 1 0.2297269
Post:Adult 0.9421 1 0.3317506
Post:Pre 2.7315 1 0.0983886 .
Adult:Pre 12.0664 1 0.0005134 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
I then use lsmeans
to compute the pairwise comparisons from the contrasts.
Pre_Adult_PC <- lsmeans(lm13, ~Pre*Adult)
Pre_Adult_PC
Pre Adult lsmean SE df asymp.LCL asymp.UCL
C C -0.3979339 0.1174724 NA -0.6281756 -0.1676922
B C -0.5141986 0.1183483 NA -0.7461569 -0.2822402
C F- -0.5248242 0.1196266 NA -0.7592880 -0.2903604
B F- -0.3726527 0.1176916 NA -0.6033240 -0.1419813
Results are averaged over the levels of: Post, Age
Results are given on the log (not the response) scale.
Confidence level used: 0.95
contrast(Pre_Adult_PC, alpha=0.05, method="pairwise", adjust=NULL)
contrast estimate SE df z.ratio p.value
C,C - B,C 0.11626466 0.05302493 NA 2.193 0.0283
C,C - C,F- 0.12689028 0.05373435 NA 2.361 0.0182
C,C - B,F- -0.02528124 0.05236930 NA -0.483 0.6293
B,C - C,F- 0.01062562 0.05679222 NA 0.187 0.8516
B,C - B,F- -0.14154590 0.05483311 NA -2.581 0.0098
C,F- - B,F- -0.15217152 0.05686818 NA -2.676 0.0075
I read that for mixed models where the distribution is not normal, Tukey's HSD isn't advised. This would appear to be true given how highly significant the interaction between Pre and Adult is. When I put the Tukey's HSD adjustment in, I lose nearly all the significant pairwise comparisons:
contrast(Pre_Adult_PC, alpha=0.05, method="pairwise", adjust="Tukey")
contrast estimate SE df z.ratio p.value
C,C - B,C 0.11626466 0.05302493 NA 2.193 0.1252
C,C - C,F- 0.12689028 0.05373435 NA 2.361 0.0847
C,C - B,F- -0.02528124 0.05236930 NA -0.483 0.9629
B,C - C,F- 0.01062562 0.05679222 NA 0.187 0.9977
B,C - B,F- -0.14154590 0.05483311 NA -2.581 0.0484
C,F- - B,F- -0.15217152 0.05686818 NA -2.676 0.0374
So for the time being I've specified the adjustment as NULL (I've looked at SIDAK adjustment too but still lose these signficant pariwise comparisons). Looking at my data visually, the data appear to match the stats when the NULL adjustment is applied.
I would like some advice as to whether or not I should report these contrasts and if so how I report them, considering there is no post-hoc adjustment. ie. do I say: "all pariwise comparisons were computed from the contrasts between factors using lsmeans
package".
Any help would be greatly appreciated. I hope I'm going about this the right way!
Thanks!
Dave
Best Answer
The changes in p-values you are showing don't seem unusual. Essentially, in your case, you need a p-value of < 0.01 before adjustment to remain < 0.05 after adjustment.
In my opinion, whether or not you apply a p-value adjustment depends on how conservative you want to be with the comparisons of groups. If you want to maximize the chances of identifying potential "real" differences, you would apply no adjustment. If you wish to be more conservative, and avoid "false positive" differences, you would apply an adjustment. If you want to be really conservative, use Bonferroni. If you want to be less conservative, use a less conservative adjustment.
(I know some statisticians who would want to pull their hair out with my language here, but I'm trying to keep the point simple.)
For me, that's all there is: It's up to you how you want to balance "false positives" and "false negatives". There are situations where it makes sense to be conservative and situations where it makes sense to be more liberal.