Solved – Cohen’s Kappa using (irr) and kappa2() outputs NaN

agreement-statisticscohens-kappar

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2().

For the following item (example), the responses between the two raters have a very high percent agreement but my kappa statistic ends up as NaN.

For other items I am analyzing with kappa2(), I do get kappa values that seem to make sense – and I understand that it is very common for Cohen's Kappa to be much lower than the percent agreement (sometimes extremely so). I also understand that normally, if the kappa statistic is 0, then any agreement or disagreement between raters is due to chance alone.

Q1) But what about the case where kappa outputs as "NaN"? What does this mean?

Q2) Is there any general concern that, despite multiple possible categories (1-5 for example), raters only use 1 or 2 of the possible categories (i.e., because the data are relatively homogeneous)?

The variable being rated is categorical, with possible responses of 1, 2, 3, or 4 – obviously both raters always rated as 1, except in cases of NA [where data could not be coded])

item1.rater1<-c(1,1,NA,1,1,NA,1,1,1,1,1,1,1,1)
item1.rater2<-c(1,NA,NA,1,1,NA,1,1,1,1,1,NA,NA,1)
item1<-as.matrix(cbind(item1.rater1,item1.rater2))
kappa2(item1[,c(1,2)])

Cohen's Kappa for 2 Raters (Weights: unweighted)

Subjects = 9 
  Raters = 2 
   Kappa = NaN 

       z = NaN 
 p-value = NaN

Best Answer

In this data, your two raters agreed 100% of the time (the function excludes all rows that include an NA). They both individually have a 100% estimated chance of selecting category "1". Thus, the probability of agreement by chance is $p_e = 1*1 = 1$. Notice that the denominator of Kappa includes $1-p_e$. Thus you are trying to divide by zero. In R, since the numerator also happens to be zero, you end up with $0/0 = NaN$.

It seems to me this is an edge case that the irr package developers might want address in their next version of the package.

Related Question