Solved – Is p-value also the false discovery rate

false-discovery-ratehypothesis testingp-valuestatistical significancetype-i-and-ii-errors

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) it states

For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery is, by definition, 0.05.

My question: I always thought false discovery is Type I error, which is equal to the chosen significance levels in most tests. P-value is the value calculated from the sample. Indeed, Wikipedia states

The p-value should not be confused with the significance level $\alpha$ in the Neyman–Pearson approach or the Type I error rate [false positive rate]"

So why does the linked article claim that Type I error rate is given by the p-value?

Best Answer

Your false discovery rate not only depends on the p-value threshold, but also on the truth. In fact, if your null hypothesis is in reality wrong it is impossible for you to make a false discovery.

Maybe it's helpful to think of it like that: the p-value threshold is the probability of making false discoveries when there are no true discoveries to be make (or to put it differently, if the null hypothesis is true).

Basically,

Type 1 Error Rate = "Probability of rejecting the null if it's true" = p-value threshold

and

Type 1 Error Rate = False Discovery Rate IF the null hypothesis is true

is correct, but note the conditional on the true null. The false discovery rate does not have this conditional and thereby depends on the unknown truth of how many of your null hypotheses are actually correct or not.

It's also worthwhile to consider that when you control the false discovery rate using a procedure like Benjamini-Hochberg you are never able to estimate the actually false discovery rate, instead you control it by estimating an upper bound. To do more you would actually need to be able to detect that the null hypothesis is true using statistics, when you can only detect violations of a certain magnitude (depending on the power of your test).

Related Question