Solved – Limitations of Cohen’s kappa for sparse data

cohens-kappasparse

Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, lots of values < 5).

Best Answer

Cohens Kappa is known to have limitations for skewed datasets.

Quoting an example from here

Consider following matrix :

+---+----+
| 1 | 6  |
+---+----+
| 9 | 84 |
+---+----+

The above example has an observed agreement of 0.85 but the Cohen' Kappa is 0.04.

The solution suggested in this article is to report two separate agreement metrics for positive and negative classes.