I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement.
Can anyone advise on how to calculate a kappa statistic that is adjusted to account for prevalence of the object of measurement?
Best Answer
Kappa is regarded as a measure of chance-adjusted agreement, calculated as $\frac{{{p_{obs}} - {p_{exp}}}}{{1 - {p_{exp}}}}$ where ${p_{obs}} = \sum\limits_{i = 1}^k {{p_{ii}}} $ and ${p_{exp}} = \sum\limits_{i = 1}^k {{p_{i + }}{p_{ + i}}} $ ($p_{i+}$ and $p_{+i}$ are the marginal totals). Essentially, it is a measure of the agreement that is greater than expected by chance.
Where the prevalence of one of the categories is high, chance agreement will be high and kappa can have unexpectedly low values. To adjust for this, and for the bias of kappa (I don't have sources nearby to refresh my memory to be able to write about the bias), the prevalence and bias adjusted kappa (PABAK) can be used. This can be calculated as $\frac{{k{p_{obs}} - 1}}{{k - 1}}$ where $k$ is the number of categories.
There is some difference of opinion on whether to use PABAK. Some commentators believe the prevalence and bias effects are important in themselves so kappa should be reported, together with measures of prevalence and bias. Others like the convenience of a single number. I chose to report both PABAK and kappa when I needed to use kappa.