Inter-rater agreement in Python (Cohen's Kappa)
python, rating, statistics
Solution
Cohen's kappa was introduced in scikit-learn 0.17:
sklearn.metrics.cohen_kappa_score(y1, y2, labels=None, weights=None)
Example:
from sklearn.metrics import cohen_kappa_score
labeler1 = [2, 0, 2, 2, 0, 1]
labeler2 = [0, 0, 2, 2, 0, 2]
cohen_kappa_score(labeler1, labeler2)
As a reminder, from {1}:
References:
- {1} Viera, Anthony J., and Joanne M. Garrett. "Understanding interobserver agreement: the kappa statistic." Fam Med 37, no. 5 (2005): 360-363. https://www.ncbi.nlm.nih.gov/pubmed/15883903:
Problem
I have ratings for 60 cases by 3 raters. These are in lists organized by document - the first element refers to the rating of the first document, the second of the second document, and so on: ``` rater1 = [-8,-7,8,6,2,-5,...] rater2 = [-3,-5,3,3,2,-2,...] rater3 = [-4,-2,1,0,0,-2,...] ``` Is there a python implementation of Cohen's Kappa somewhere? I couldn't find anything in numpy or scipy, and nothing here on stackoverflow, but maybe I missed it? This is quite a common statistic, so I'm surprised I can't find it for a language like Python.