Solved – Rank correlation statistics comparison

ginikendall-taurocspearman-rho

I am trying to understand the relative behavior of the following rank correlation statistics:

  1. Spearman coefficient
  2. Kendall Tau / Concordance percentage
  3. Normalized Gini coefficient (area under curve of percentage captured versus percentage observations)
  4. Normalized Area under ROC curve (for binary classifiers)

I don't believe any of these are functionally related to the others. The accepted answer here references this paper and Spearman and Kendall are highly correlated (as one would expect).

Are there good intuitions behind/discussions of relative (across datasets) or absolute (for a given dataset) differences for (any pair of) these measures?

Best Answer

The proposed question is rather complicated. As analystic already pointed out, I don't think all these measures can be compared straightforwardly, because rank correlation coefficients, Gini coefficient, and AUC (area under ROC curve) are generally defined on different domains.

However, there is a very close relation between Kendall's $\tau$ and Spearman's $\rho$, the two rank correlation coefficients in the list. While the paper cohoz mentioned has demonstrated their relation empirically (Figure 3), this relation can actually be quantified theoretically. Let $\pi$ and $\sigma$ be two rankings, and $\pi(i)$ and $\sigma(i)$ be the ranks of item $i$ in $\pi$ and $\sigma$, respectively. The Kendall distance and Spearman distance between $\pi$ and $\sigma$ are defined as follows: $$ K(\pi,\sigma) = \# \lbrace \; (i,j) \, \vert \, \pi(i)>\pi(j) \text{ and } \sigma(i)<\sigma(j) \; \rbrace $$ $$ S(\pi,\sigma) = \sum_i \left( \pi(i) - \sigma(i)\right)^2 $$ We have the following relation between $K$ and $S$ following [Diaconis and Graham 1977]: $$ \frac{1}{\sqrt{n}}K(\pi,\sigma) \le S(\pi,\sigma) \le 2K(\pi,\sigma) $$ Because the rank correlation coefficients are just the normalization of the rank distances to the interval $[-1,1]$, similar inequalities can be easily derived between $\tau$ and $\rho$. In the statistical ranking literature, results are mostly represented in terms of distances rather than coefficients.

Two more things:

  1. The rankings $\pi$ and $\sigma$ must be complete rankings in order to make this inequality hold. That is, they cannot be partial rankings.
  2. In case one is interested in $\tau$ and $\rho$ defined not only on rankings but on continuous random variables, the situation is more involved. Here is a related paper by Fredicks and Nelsen.
Related Question