Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to get a p-value of 0.05?
Assuming a power of 0.8, if I were to take 100 samples of drugs with 10 percent that actually work, then 8 will show up as significant, and with an alpha of .05, about 5 will be type I errors. Thus the FDR is 38% or 5/13.
However, if I happen to get a p-value super extreme like 0.001, will my FDR change? If so, why does it, considering that alpha and percent of working drugs seem to dictate the FDR, rather than the p-value of a sample.
Best Answer
The standard definition of false discovery rate (FDR) is the expected proportion of false discoveries among all discoveries. As you note and as explained here in other answers, it does not depend on the $p$-value but only on the $\alpha$.
I guess what you are thinking about, is the expected proportion of false discoveries among the discoveries with a given $p$-value, e.g. among the discoveries with $p=0.001$. Strictly speaking, this is not called "false discovery rate" anymore. However, it is a meaningful quantity.
To say it differently, false discovery rate is the probability of the null hypothesis being true given that $p\le\alpha$, i.e. $$\mathrm{FDR}=P(H_0 | p \le \alpha).$$ The quantity you seem to be thinking about is simply $P(H_0 | p)$. This latter quantity will of course depend on $p$, and will be lower for lower values of $p$.
David Colquhoun (2014) does call it "false discovery rate", see Section 10 of his paper An investigation of the false discovery rate and the misinterpretation of $p$-values, and uses simple simulation to show that the lower the $p$-value, the smaller this FDR-conditioned-on-the-p-value will be. I guess though that this terminology can be considered sloppy. Also, see here for some lengthy discussion of this paper.