Solved – Benjamini-Hochberg: choosing the False Discovery Rate / q value

multiple-comparisons

Background: the Benjamini-Hochberg procedure is a method for correcting for multiple p-values. (If you set p<.05, and you do 20 tests, then one test is likely to be a false positive. The B-H procedure corrects for this.)

When using this procedure, not only do you set a p value (usually .05), you also set a q value – the False Discovery Rate (or FDR). This does not necessarily need to be .05.

This CrossValidated thread discusses the correctness of choosing the q value after viewing your data, but it does not answer my question.

My question: what is usually considered a reasonable q value? I've seen everything from .05 to .20 used in literature, but I have found no guidance on what are the highest values that may be considered "conservative" and what is considered a stretch.

Best Answer

FDRs are commonly much greater than p-values, and recall the lowest FDR value represents a list of features and not one feature, like a p-value does. So if the lowest value of FDR is for example 0.15 for a list of 15 features(genes), then at the very least you have to publish the list and state that the FDR is 0.15. If reviewers expect to see lower FDR's then there's not much of an alternative. Only lower p-values can drive FDR lower. If you can generate a list of 10-15 or 20-30 features whose FDR is 0.05, then you should have no problem publishing this in the peer-reviewed literature. However, FDR values of 0.1, 0.15, and 0.2 and greater are frowned upon -- i.e. are not considered "good." This does not mean however, that you can't publish a list of features when FDR is not 0.05 or is greater than 0.05 -- it's a matter of the personal-experience-based FDR threshold assumed by the reviewer, laboratory, or journal.

Related Question