cohens-kappa
I have two raters who agree on 93% on the cases (two options: yes or no). However, when calculating cohens kappa through crosstabs in spss I get really strange outcomes like -0.42 with a sig. of 0.677.
How can such a high agreement in percentage result in such a strange kappa?
I don´t get it.
In this data, your two raters agreed 100% of the time (the function excludes all rows that include an NA). They both individually have a 100% estimated chance of selecting category "1". Thus, the probability of agreement by chance is $p_e = 1*1 = 1$. Notice that the denominator of Kappa includes $1-p_e$. Thus you are trying to divide by zero. In R, since the numerator also happens to be zero, you end up with $0/0 = NaN$.
It seems to me this is an edge case that the irr package developers might want address in their next version of the package.
I find this document taken from this refresher on Bivariate Statistics Hand-Computation Cache from the University of Nebraska–Lincoln to be clear:
Best Answer
Suppose there are 100 ratings. In 98 of them, both raters say "Yes". In 1, rater A says "no" and rater B says "yes"; in 1, just the opposite happens. Now kappa is near 0 (actually slightly below 0) but agreement is 98%. If you increase the '1's you lower both kappa and agreement.
Kappa is, in a way, designed to do this. It is intended to reduce the influence of matching that isn't indicative of anything.