Solved – Weighted Kendall tau rank correlation coefficient

kendall-taumetricrankingranks

I need to use a measure to compare two rankings:

Consider the ground truth as 1,2,3,4,5,6. Let two rankings be R1 and R2.

R1: 1,2,3,6,4,5
R2: 1,4,2,3,5,6

From Wikipedia Kendall's tau ranking correlation coefficient is given by:
$$
\frac{\text{Number of Concordant Pairs – Number of Discordant Pairs}}{(n(n-1)/2) }
$$
Calculation of Kendall's tau for R1:

Number of Concordant Pairs = 15
Number of Discordant Pairs = 2

Similarly for R2: 

Number of Concordant Pairs = 15
Number of Discordant Pairs = 2

Thus, although the Kendall's correlation coefficient is same for both the lists, I want to penalise R2 more as the differences in position is towards the head than tail. Thus, along with the ranking, I also want to take into consideration the position. Is there a well defined measure for this? How can this be done?

Best Answer

I don't have commenting privileges, so I will attempt an answer here. Perhaps your original question is unclear, but here are answers depending on your exact meaning:

"I want to penalise R2 more as the differences in position is towards the head than tail. Thus, along with the ranking, I also want to take into consideration the position."

  • If you want to penalize R2 because it moved too far towards position 1, then despite your response to another answer, you do care about relevance. In other words, if errors too far towards the head or towards the tail matter, than relevance-based ranking is what you are looking for. The other answer's suggestion of Discounted cumulative gain is a good choice.
  • Alternatively, I don't know if you think that in R2 that there was a bigger absolute change or jump in the ranking, for which you want to give a penalty. In fact, the difference in both cases is -2: In R1, 6 moved -2 to rank 4; in R2, 4 moved -2 to 2. Thus, Kendall's tau is identical, because tau only cares about how much difference there is, not where exactly the jump occurred. If for instance, there had only been a jump in 1 (e.g. if R3 were to be 2,1,3,4,5,6), then tau would have a larger value (indicating more concordance). If that's the case, then Kendall's tau might be just what you need.
Related Question