Suppose that $k$ experts are each asked to rank a set of $n$ objects in order or preference. Let allow ties in the rankings.
John Kemeny and Laurie Snell in their 1962 year book "Mathematical models in the Social Sciences" propose to solve next problem:
PROJECT $1$.
Develop a measure of the reliability of a consensus ranking
by $k$ experts. For example, this may be based on the largest possible change that can be brought about by changing the ranking of a single expert. (Attention must be paid to the possibility of multiple consensus rankings.) Prove some theorems concerning the most and least reliable consensuses possible for a given $k$.
The book gives notation for rankings and method for rankings aggregation (i.e. getting one "collective" ranking from many "individuals"). But no answer given for the problem above.
First, I thought about Kendall's $W$ coefficient of concordance, but it looks like it doesn't suit. Any ideas are welcome!
Best Answer
As a measure of reliability of a consensus ranking by $k$ experts you can use the $\tau$-extended by Emond and Mason. For extensive explanations take a look to: http://onlinelibrary.wiley.com/doi/10.1002/mcda.313/full or http://www.sciencedirect.com/science/article/pii/S0377221715008048