Solved – Significance test on the difference of Spearman’s correlation coefficient

hypothesis testingspearman-rhostatistical significance

(Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.)

I do not know how to find out whether or not the difference between two Spearman's correlations is statistically significant. I would like to know how to find it out.

The reason I wanted to find out is that in the following paper: Wikipedia-based Semantic Interpretation for Natural Language Processing, by Gabrilovich and Markovitch (Journal of Artificial Intelligence Research 34 (2009) 443-498).

In Table 2 (p. 457), the authors are showing that their method (ESA-Wikipedia) achieves a higher and statistically significant Spearman's correlation than other methods, and I would like to do the same to show that my method is better than previous methods for some problem.

I do not know how they calculated statistical significance, and I would like to know. The author of the paper did state that Spearman's rank correlation was treated as Pearson's correlation. I am not sure if that is the right way to do it. I have two Spearman's correlations and I would like to know ifthe difference between them is statistically significant or not.

I am aware that web sites, such as http://faculty.vassar.edu/lowry/rdiff.html, provide online calculator for obtaining the difference between two Pearson's correlations. I am unable to find a similar online calculator for the difference between two Spearman's correlations.

A solution from the link provided by Peter Flom

NOTE: The procedures only support the Spearman's correlations that are under 0.6.

  1. Let $z_A$ = the Fisher transform of the observed correlation of set $A$,
    $z_B$ = the Fisher transform of the observed correlation of set $B$.

  2. For $i = 1,\dots,n$, let $y_{A_i} = nz_A- (n – 1)z_{A'i}$, where $z_{A'i}$ is the Fisher
    transform of set $A$ of the one-left-out correlation obtained by deleting
    $(x_i,y_i)$, re-ranking, and re-computing the correlation. (Each $z_{A'i}$
    is based on $n-1$ pairs; each deletion is temporary, for that i only,
    not permanent.) Repeat for set $B$.

  3. $\bar y_A = \sum y_{A_i}/n$ is the jackknifed Fisher transform.
    Repeat for set $B$.

  4. $v_{\bar y_A} = \sum (y_{A_i}-\bar y_A)^2 /(n(n-1))$ is the variance of $\bar y_A$.
    Repeat for set $B$.

  5. Use a heteroscedastic (Welch-Satterthwaite) $t$-test to compare the two jackknifed estimates:

$$ t = \frac{\bar y_A – \bar y_B}{\sqrt{v_{\bar y_A} + v_{\bar y_B}}},\quad \text{df}=\frac{(v_{\bar y_A} + v_{\bar y_B})^2}{\frac{v_{\bar y_A}^2}{n_A-1}+\frac{v_{\bar y_B}^2}{n_B-1}}$$
where $n_A$ and $n_B$ are the number of samples of set $A$ and $B$ respectively.

Before first edit

I have got a human-rated set of ranking (HUMAN-RANKING), a set of ranking generated by the presently used, popular method (PRESENT-RANKING), and finally a set of ranking generated by my purposed method (MY-RANKING).

I calculated the Spearman's correlation between HUMAN-RANKING and PRESENT-RANKING. Let me call this: HUMAN-PRESENT-SPEARMAN.

I then found out the Spearman's correlation between HUMAN-RANKING and MY-RANKING. Let me call this: HUMAN-MY-SPEARMAN.

How can I find out if the difference between HUMAN-MY-SPEARMAN and HUMAN-PRESENT-SPEARMAN is statistically significant?

Best Answer

The paper you cite explains the method in the following terms:

[...] we show the statistical significance of the difference between the performance of ESA-Wikipedia (March 26, 2006) version) and that of other algorithms by using Fisher's z-transformation (Press, Teukolsky,Vetterling, & Flannery, Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, 1997, Section 14.5).

I suggest you follow that reference, or have a look at the Wikipedia page on the Spearman coefficient for details.

Related Question