I have:
Condition #1 : 5 independent objects (1 object = a time series)
Condition #2 : 5 independent objects
My GOAL: test H0 that using condition #1 gives different results from using condition #2
I have a pairwise test statistics to test if one pair of objects are different. In total, I can calculate 5! pairwise comparisons => 120 p-values
Now I use Benjamini-Hochberg procedure to calculate adjusted p-values in R:
p.adjust(pvalues, method="BH")
(I can use Benjamini and Yekutieli instead for dependence, but lets skip this for now)
As far as I understand, this gives me a set of q-values, that is
"the adjusted p-value of an individual hypothesis is the lowest level
of FDR for which the hypothesis is first included in the set of
rejected hypotheses." (Reiner et.al, Bioinformatics, 2003).
Now I have 120 adjusted p.values, and I reject the null for all cases when p < 0.05. This would control the false discovery rate at 5%.
Can I say that "condition #1 gives different results from using condition #2" if number of rejected nulls is more than 120*5% = 6? In other words, since I expect 5% of false positives, can I say that the results are significant if I see more than 5% positives?
Best Answer
Firstly we are going to formalize the hypotheses that you are testing
$$H_0 = \{ \text{using condition #1 gives the same results as using condition #2} \}$$
versus
$$H_1 = \{ \text{using condition #1 gives different results than using condition #2} \}$$
(I corrected your $H_0$, I think you want to formulate it like this)
Then you can formulate your hypothesis in terms of smaller hypothesis
$$H_0 = \cap_{i,j \in \{1,2,3,4,5\}} \{ \text{time series $i$ from condition 1 and time series $j$}$$
$$\quad \quad \quad \quad \text{ from condition 2 are comparable}\}$$
Is that correct?
Further, in this formulation there are only 5*5=25 smaller hypotheses to test, not 120...
With respect to our question,I would say no because significance implies that you are controling the probability for an type 1 error. Instead you are controling the FDR which is another concept.