I have done 350 independent tests. I need to correct them for the multiple comparisons. I am using the Benjamini-Hochberg (BH) procedure to control FDR. However choosing the Q (the level at which we want to control the FDR) beforehand could be restrictive (say e.g. I find no discovery for Q=0.05 and 40 for Q=0.1: it would be nice tell something about those 36 supposedly true discoveries made with Q=0.1 instead of reporting no discovery).
I have two questions:
-
If I'm doing an exploratory analysis, is it a wrong procedure to have a look at the number of discoveries for varying levels of Q instead of fixing it a priori? If I then need to choose just one Q value I may choose, e.g. the more conservative one that still gives me some true discovery or the one that gives the maximum number of true discovery, etc.
-
Am I doing again multiple comparisons in checking different levels of Q? I don't think so since I'm actually not testing new hypothesis but just controlling the FDR to different levels: the original hypothesis keep untouched, what is changed are the criteria in the BH procedure.
Best Answer
Yes, these things are done quite often, at least in genetics. To address your specific points:
This is quite commonly done, and I have personally reported results this way. Though, make sure to make it clear at what level of FDR you are reporting. Think of it akin to "marginal" significance; people may find it interesting, but they have to know what they're looking at.
No, you would not have to do multiple corrections for this, as you mentioned you're simply manipulating the $P$-values in a different way and not changing anything.
I would also look further into what Dr. Motulsky has mentioned above, if I were you. Reporting the $q$ value is a very common and useful metric.