I was wondering which method among "fdr","none","bonferroni" is the
most accurate one to calculate the adjusted p-value for multiple
comparison tests. Since when I define
summary(glht(fit1.2,linfct=mcp(Ethnic.Sex=cMat)),test=adjusted("none"))
I will have several significant p-values, but when I use Bonferroni and fdr the p-values are not significant. So any advice would be highly appreciated?
Best Answer
The idea behind these corrections is that the the chance of getting a false positive for several test say 20 at an $\alpha=0.05$ is 1 in 20. Out of 20 comparisons one has a strong possibility of being a false positive.
"none" says don't correct the test and use $\alpha = 0.05$.
The Bonferroni correction is the most conservative measure of the three you mentioned where the adjusted significance level is $\alpha/m$ where $m$ is the number of hypotheses.
FDR or False Discovery Rate correction in R is an alias for the 'BH' Benjamini & Hochberg correction. This method ranks the unadjusted p-values and the new pvalue is less than or equal to $\alpha * rank / m$ where $m$ is the number of hypotheses.
As a rule of thumb, in terms of a false positive I would order them as follows (lowest to highest):
bfr $<$ fdr $<$ none
You can do more research into each and learn about the family wise error rate vs the false discovery rate.
Question and answer on Stackexchange about the bonferroni correction vs Benjamini-Hochberg correction as the number of comparisons increase