Hypothesis Testing – Why Complementary p-values Do Not Cancel in Combining Methods

combining-p-valueshypothesis testingmeta-analysis

What is this about?

Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ values $p_1=x$ and $p_2=1-x$.
I have no further a priori knowledge that makes a distinction between the datasets, e.g., to weigh them.
In this case, I do not learn anything from my tests as to whether my research hypothesis is true or not.
If I tested the opposite research hypothesis, I would obtain the same results (just in different order).
Therefore the accurate result for combining these $p$ values is $p_\text{comb} = \frac{1}{2}$.

However many methods for combining $p$ values do not treat $p$ values and their complements in a symmetrical fashion and produce implausible results.
For example for $x=0.01$:

Fisher’s method: $p_\text{comb} = 0.06$
Pearson’s method: $p_\text{comb} = 0.94$
Tippett’s method: $p_\text{comb} = 0.02$
Simes’ method: $p_\text{comb} = 0.02$

By contrast, Stouffer’s method, Mudholkar’s and George’s method, as well as Edgington’s method are symmetric (as described above) and produce $p_\text{comb} = \frac{1}{2}$.

Obviously, this problem extends beyond the above simple example and can lead to many clear false positives in many cases. I could probably produce datasets where two opposing one-side research hypotheses are both highly significant.

My question

I consider it a serious flaw of a combining method if complementary $p$ values do not cancel each other.
However, I fail to find this issue mentioned in the literature on combining $p$ values.
To give just one example, the paper Choosing Between Methods of Combining $p$-values does not mention it as far as I can tell.
In fact, the only mention I have found so far is here on this site.

So, what am I missing?

Is there literature on this (and I failed to find it)?
Is my argument somehow flawed and I am overestimating the importance of this?
Is this generally accepted, but just not documented?

Best Answer

I am not sure when the methods were first systematically compared but Loughin in a paper entitled "A systematic comparison of methods for combining p-values from independent tests" available here compared some of them. He does refer to previous work which discussed the issue you raise. Some other methods are compared in my R package metap and the comparisons set out in one of the vignettes. (Apologies for the self-promotion.)

I am not an expert on modern genetics but I believe that there the interest is in which tests display the signal and so a method which does not cancel is quite acceptable and indeed even preferred.

As you suggest plotting the $p$-values should really be obligatory as then it becomes clear that there are extreme values in both directions which should lead the investigator to question their theory. I am not aware of substantial sources recommending this though.

Related Solutions

Hypothesis Testing – Why Not Just Average When Combining p-values?

You can perfectly use the mean $p$-value.

Fisher’s method set sets a threshold $s_\alpha$ on $-2 \sum_{i=1}^n \log p_i$, such that if the null hypothesis $H_0$ : all $p$-values are $\sim U(0,1)$ holds, then $-2 \sum_i \log p_i$ exceeds $s_\alpha$ with probability $\alpha$. $H_0$ is rejected when this happens.

Usually one takes $\alpha = 0.05$ and $s_\alpha$ is given by a quantile of $\chi^2(2n)$. Equivalently, one can work on the product $\prod_i p_i$ which is lower than $e^{-s_\alpha/2}$ with probability $\alpha$. Here is, for $n=2$, a graph showing the rejection zone (in red) (here we use $s_\alpha = 9.49$. The rejection zone has area = 0.05.

Fisher

Now you can chose to work on ${1\over n} \sum_{i=1}^n p_i$ instead, or equivalently on $\sum_i p_i$. You just need to find a threshold $t_\alpha$ such that $\sum p_i$ is below $t_\alpha$ with probability $\alpha$; exact computation $t_\alpha$ is tedious – for $n$ big enough you can rely on central limit theorem; for $n = 2$, $t_\alpha = (2\alpha)^{1\over 2}$. The following graph shows the rejection zone (area = 0.05 again).

sum of p values

As you can imagine, many other shapes for the rejection zone are possibles, and have been proposed. It is not a priori clear which is better – i.e. which has greater power.

Let‘s assume that $p_1$, $p_2$ come from a bilateral $z$-test with non-centrality parameter 1 :

> p1 <- pchisq( rnorm(1e4, 1, 1)**2, df=1, lower.tail=FALSE )
> p2 <- pchisq( rnorm(1e4, 1, 1)**2, df=1, lower.tail=FALSE )

Let's have a look on the scatterplot with in red the points for which the null hypothesis is rejected.

Scatterplot

The power of Fisher’s product method is approximately

> sum(p1*p2<exp(-9.49/2))/1e4
[1] 0.2245

The power of the method based on the sum of $p$-values is approximately

> sum(p1+p2<sqrt(0.1))/1e4
[1] 0.1963

So Fisher’s method wins – at least in this case.

Solved – How to achieve a two-sided combined p-value using Fisher’s method

As you have found out Fisher's method does not cancel values in opposite directions. The same is true of Tippett's method (which uses the minimum $p$). However the good news is that Stouffer's method (which $z$-transforms the $p$) and Edgington's method (which sums the $p$) do cancel so you could use one of them instead.

If you go to the page for the metap R package here and look at the vignette you will find some worked examples. Disclaimer: I am the author of that package.

Edit to add comments on directionality

The null hypothesis $H_0$ is well defined, that all $p_i$ have a uniform distribution on the unit interval. There are two classes of alternative hypothesis

$H_A$: all $p_i$ have the same (unknown) non--uniform, non--increasing density,
$H_B$: at least one $p_i$ has an (unknown) non--uniform, non--increasing density.

So these are basically omnibus tests where there is no obvious directionality built in.

The lack of a natural alternative hypothesis may account for the number of methods available and their differing behaviour.

Given that there is no obvious directionality built in what people do in a substantive application is take a look at the data and if they see $p$-values piling up near 0 assume the effect is in that direction and if near 1 the contrary direction. I suppose there is nothing to stop people observing piling up at both ends and interpreting accordingly. Another course of action would be to use one of the methods like Fisher's which does not cancel and then perform it on the complement of the $p$-values where it would be sensitive to piling up at the other end. I am not aware of any settings in which these have been applied but that may just be me.