Wikipedia:
"In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests."
"The false discovery rate (FDR) is one way of conceptualizing the rate of type I errors in null hypothesis testing when conducting multiple comparisons."
I don't understand the difference between these two concepts. How do they not mean the same?
Perhaps you can help me by further elaborating the following example:
Say the probability for an unbiased coin to substantially deviate from a 50/50 head/tail-distribution in a sequence of 1,000 tosses is 0.001.
If I want to find out if one coin is biased I throw it 1,000 times and if it shows heads ~500 times I can be quite sure it is not biased.
However if I throw a million coins 1,000 times and deem those biased who don't show a 50/50-distribution of heads and tails, I will categorize unbiased coins as biased, because the probability of an unbiased coin showing deviating from the 50/50-distribution is multiplied by the number of coins (1 million).
Thus from a set of one million unbiased coins, I have to expect about 1,000,000*0.001=1,000 coins to deviate substantially from the 50% tails, 50% heads-distribution.
As far as I understood this is multiple hypotheses testing (synonymous: multiple comparisons?) as I am testing the hypothesis "coin is unbiased" a million times, and the false discovery rate FDR is 1,000 in this example.
But what, then, is the FWER (family wise error rate)?
Best Answer
Part of the reason you're confused may be that you are considering the special case that all null hypotheses are true (i.e. m = m0). When all null hypotheses are true, the FWER and FDR are indeed the same. For m independent tests of true null hypotheses, FDR = FWER = 1-(1-alpha)^m.
The difference comes when some null hypotheses are true and some null hypotheses are false. In that case, the FDR tells you the expected proportion of significant tests (not of all tests) that will be Type I errors. Computing the FDR is then not as simple, because it depends on the proportion of null hypotheses that are false and also on power (the probabilities of significance for the tests of the false null hypotheses).
Neither FWER nor FDR can ever be greater than 1. The value of 1,000 that you computed is a different error rate called the per-family error rate: PFER = alpha*m.