Solved – Why are the q-values equal to 1 after adjusting for multiple testing via the FDR

anovafalse-discovery-ratemultiple-comparisons

I am testing for differential expression of micro RNAs (miRNA), and looking for changes in the mean of a continuous variable under a categorical variable that takes values 0 and 1. There are many different miRNA features, and I test each one in turn for a change in its mean. I performed a false-discovery rate (FDR) adjustment to account for these multiple comparisons. My q-values (reported by the false discovery rate procedure) turn out to be all equal to 1.

Can someone explain why this happened, and what it means?

Best Answer

Two points

Point the First
If you are using something like R's p.adjust() to calculate $q$ values, then the 1 values simply indicate not rejected at any level of FDR. $q$-values are actually a little problematic to interpret directly, since they have a subtle mathematical artifice and because they do not communicate the step-wise nature of the FDR adjustment process (and one cannot make FDR rejection decisions based on $q$-values alone). Backing up to a single two-sided hypothesis can help illustrate why:

Reject $H_{0}$ if $p \le \alpha/2$. So for $\alpha = 0.05$, we would reject $H_{0}$ if $p \le 0.025$. Alternately, we could express this same rejection criterion as reject $H_{0}$ if $2p \le \alpha$. The first expression perhaps emphasizes the meaning of $p$, and the second emphasizes the meaning of $\alpha$.

If we think about the Bonferroni method (FWER, not FDR), we can see that we have two way to express the rejection criterion given $m$ number of comparisons:

Reject $H_{0}$ if $p \le \frac{\alpha/2}{m}$, or

Reject $H_{0}$ if $2mp \le \alpha$.

That $2mp$ is an 'adjusted $p$-value', sometimes called a '$q$-value'.

(I suppose there's also a third way: reject $H_{0}$ if $mp \le \alpha/2$.)

But look: $2mp$ is $>1$ when $p>.5/m$, which is quite possible. Unfortunately $p$ (or $q$) is supposed to be a probability which means that it's value is strictly bounded by zero and one inclusive. So many folks, and many statistical software authors will take an expression like $q = mp$, and replace it with $q=\max(1,mp)$. The same applies to FDR (whether using the Benjamini-Hochberg or Benjamini-Yekutieli method)... the adjustments are more complicated than the Bonferroni, but they cap the $q$-value results at 1.

In a way, I suspect that this implies that expressing such adjustments as adjustments of rejection levels, rather than adjustments to $p$-values is a little more coherent, because the artifice of $\max(1,f(p,i))$ does not apply.

Point the Second
We can't tell for sure because you have not provided, for example, your vector of $p$-values, but the likelihood is that your $p$-values are all too high, and that you are not achieving significance.