Intercoder reliability: Cohen's kappa and Krippendorff's alpha
Two or more coders coded the same units – to what extent do they agree beyond chance? Raw agreement, Cohen's κ for each pair with a confidence interval, and Krippendorff's α for any number of coders, with missing values, with a nominal, ordinal or interval metric – with a bootstrap confidence interval; weighted kappa for ordinal scales and Fleiss' kappa for multiple coders.
Loading the data
Paste columns directly from the sheet (including the header row) – select them in Excel, copy, and paste here:
Selecting the coders and the metric
When are intercoder agreement measures used?
When human beings are the measuring instrument. An open-ended survey response classified by topic, a post marked as "hostile" or "neutral", an interview segment that received a code from a codebook, a clinical diagnosis from a list of categories – in all of these the number that enters the analysis is a judgment. Assessing intercoder reliability asks whether another coder, given the same instructions, would arrive at the same codes – and without a positive answer, it is impossible to know whether the findings describe the text or the person who read it. Cohen (1960) put this in the image of a "two-legged meter": the coders are parallel forms of the same instrument, and the question is how much they agree. Krippendorff (2019, chapter 11) adds the other side: reliability is a necessary condition for validity, not a guarantee of it – two coders can agree and be wrong together.
This page is the sixth step in the workflow of the Hebrew NLP page: after manual coding of a sample (by two people, or by a person and a model), the agreement is measured here – and only if it is sufficient, move on to frequencies, chi-square and correlations.
Two coefficients, two questions: Cohen's κ and Krippendorff's α
Raw agreement (percent agreement) – the proportion of units on which the coders agreed – is the intuitive number, and it is misleading: two coders tossing a coin will agree in 50% of cases with two categories, and in 80% if one category is very common. All the serious coefficients correct for this in the same way: (observed agreement − agreement expected by chance) / (1 − agreement expected by chance). The difference between them lies in what counts as "chance".
Cohen's κ (1960)
Two coders, nominal categories, no missing values. Chance agreement is computed from the table margins of each coder separately: if A marks "economic" 40% of the time and B 50%, the chance expectation for that category is .40 × .50. The most common coefficient in psychology and medicine, built into SPSS, with a standard error and a confidence interval. It also gives κmax – the upper bound that the margins allow.
Krippendorff's α (2019)
Any number of coders, missing values allowed, nominal / ordinal / interval / ratio metric. Chance agreement is computed from the distribution of all values together, without asking who gave them – the coders are interchangeable. The accepted coefficient in content analysis and communication research, with defined thresholds (.800 / .667). In SPSS – only through a macro.
When the coders use the categories with the same frequencies, the two coefficients are almost identical. When they do not, they diverge, and in a known direction: Krippendorff (2019, section 11.5) demonstrates two tables with the same 46 out of 100 agreements, where in one the remaining agreement is spread symmetrically and in the other concentrated on one side; Scott's π (and α) give .186 for both, whereas κ rises from .186 to .258 in the second – "it is not the agreement that grew but the ability to predict one coder's categories from those of the other", and in his view κ thereby measures something that is not reliability. Cohen (1960, p. 41) saw exactly the same property as an advantage: a coder's systematic preference for a certain category is a source of disagreement, and it should not be corrected upward. The dispute is not settled, and the page reports both: κ because readers, journals and SPSS expect it; α because it is the only one that handles three or more coders and missing values without discarding data.
How to interpret the result?
The agreement matrix (Table 1, for two coders) is the table from which everything is computed: rows – the first coder, columns – the second, the diagonal – agreement. Before looking at the coefficient, it is worth looking at the matrix: which two categories are confused with each other? That is the problem to fix in the codebook, and the coefficient alone will not reveal it.
pe – the chance agreement. The higher it is (one dominant category), the less the raw agreement counts and the smaller κ becomes. κmax (Cohen, 1960, Equation 6) – the highest kappa attainable with the given table margins; a low κmax means the coders use the categories with different frequencies, and "1 − κmax is the part of the agreement that is unattainable because of the margins" – a sign of blurred category boundaries that training can sharpen. ASE and the confidence interval – the asymptotic standard error according to Fleiss, Cohen and Everitt (1969), the same formula as SPSS (Cohen himself gave a simpler approximation in 1960, equation 7, which yields a slightly higher value). z and p test whether κ differs from zero – a test that Cohen himself called "largely worthless": that kappa is greater than zero is not the question; the question is by how much.
α – Do is the observed mean disagreement within units, De – the expected disagreement among all values in the data, and α = 1 − Do/De. In the nominal metric every disagreement is equal; in the ordinal metric a disagreement between "low" and "high" is more serious than between "low" and "medium"; in the interval metric – according to the squared difference. Therefore α of a 1-7 scale in the nominal metric is lower, sometimes much lower, than the same α in the interval metric: the difference is not an "error" but a definition of what counts as agreement.
The confidence interval of α, and q. α has no standard-error formula; Krippendorff (2019, section 11.4.2) proposes bootstrap instead: α is computed on thousands of resamples of the data, and the resulting distribution of values gives a confidence interval – and, more important to him, q, the probability that α does not pass the chosen threshold. The test "does α differ from zero" is pointless, he writes; the question is how sure we are that α is above .800 (or .667). The page implements a bootstrap of units (Efron & Tibshirani, 1993): 2,000 samples of N units with replacement, α for each sample, the 2.5 and 97.5 percentiles for the interval, and the proportion of samples below .667 and .800 as q. Two differences from Krippendorff's KALPHA macro: it samples pairs of values from the coincidence matrix rather than units, so its intervals may differ slightly; and the page uses a fixed seed, so the same data always give the same interval – the site rule: deterministic computation.
Weighted κ (Cohen, 1968; shown when the categories are numeric, three or more, and the metric is not nominal) – for ordinal scales: a disagreement between 1 and 2 is not like one between 1 and 5. Linear weights give credit by distance, quadratic weights – by squared distance; the quadratic version equals the intraclass correlation in many cases, which is why it is the most common. Both are in the table, because the choice between them is the researcher's decision, not the software's – and the one chosen is reported. Fleiss' κ (Fleiss, 1971; shown from three coders up) – the accepted extension of κ to multiple coders, although it is actually an extension of Scott's π: chance agreement is computed from the distribution of categories across all coders together, not for each coder separately. It requires an equal number of coders for every unit, so units with a missing value drop out – unlike α. When the two numbers are close, this is evidence that the missing values do not change the picture. The standard error that Fleiss (1971) gives is valid only under the hypothesis κ = 0 – good for a z test, not for a confidence interval – so the confidence interval of κF in the table comes from the same bootstrap of units as for α (on complete units only). For weighted κ, z and p are also shown: a test of κw = 0 with the standard error under H0 of Fleiss, Cohen and Everitt (1969) – the same convention by which SPSS computes Approx. T in the Symmetric Measures table, "using the asymptotic standard error assuming the null hypothesis" (Pallant, 2020, Chapter 16). This test answers a question that is almost never asked in coding research – whether the agreement is greater than zero – and the confidence interval is the useful information; it is in the table so that the output matches that of SPSS.
Two yardsticks, and what the difference is. Landis and Koch (1977, p. 165) divided the range of κ into six levels – slight, fair, moderate, substantial, almost perfect – and added in the same breath that the division is "clearly arbitrary" (clearly arbitrary) and was meant to serve as "useful benchmarks" for the example in their paper. This yardstick took root far beyond their intention. Krippendorff (2019, section 11.4.4) sets two thresholds instead: rely only on variables with α above .800, and draw only tentative conclusions between .667 and .800 – and admits that even .800 is "a rather low standard" compared with engineering and medicine. The gap is considerable: κ = .65 is "substantial agreement" for Landis and Koch and a failure to meet the threshold for Krippendorff. The page shows both, and instead of deciding it offers Krippendorff's question: what is the cost of a wrong conclusion in this study?
Assumptions and warnings
Cohen's three assumptions (1960, p. 38): the units are independent; the categories are independent, mutually exclusive and exhaustive; and the coders work independently. The third is the one that breaks in practice – two coders sitting together, or a second coder who sees the first coder's codes, produce "agreement" that is not reliability.
Reliability is a property of the variable, not of the study. Krippendorff (2019, section 11.4.4) warns against averaging α across several variables: the filing variables (date, length) are always reliable and raise the average, while the variables that matter – the ones that are hard to code – hide beneath it. Report each variable separately.
Category frequency. When one category appears in 90% of the units, even raw agreement of 90% gives a low κ – agreement on the frequent category is "expected by chance". This is a correct property of the measure (Krippendorff, 2019, section 11.4.1: a variable without variance carries no information), but it surprises researchers; if the variable is nearly constant, ask whether it is a variable at all.
Sample size for the reliability test. The confidence interval of κ shrinks with N; Krippendorff (2019, section 11.4.3) suggests setting the number of units so that the rarest category appears often enough for disagreement about it to be detectable – a sample of 30 units with a category that appears twice does not test that category.
Missing values. κ is computed only on units that both coders in the pair coded (pairwise deletion – as in SPSS); α uses every value that has at least one partner in its unit, so the N of κ and the "pairable units" of α can differ on the same data. Both are reported.
What the page does not compute. α for unitizing (unitizing; Krippendorff, 2019, Section 11.6) – when the coders also decide where a unit begins and ends; an asymptotic standard error for Fleiss' κ (Fleiss, 1971, gives it only under H0; the confidence interval – by bootstrap); and a confidence interval according to the specific algorithm of KALPHA (see above).
And how do you do it in SPSS?
Version note. The procedure has been stable since very early versions; the menu names below follow SPSS 29. Cohen (1960) and Landis and Koch (1977) predate SPSS; the standard-error formula in SPSS is that of Fleiss, Cohen and Everitt (1969), and it is the one the page computes.
Cohen's κ: Analyze › Descriptive Statistics › Crosstabs… – the first coder to Row(s), the second to Column(s), and under the Statistics… button check Kappa. The syntax:
CROSSTABS /TABLES=coder_A BY coder_B /STATISTICS=KAPPA /CELLS=COUNT.
In the Symmetric Measures table: Value = κ, Asymp. Std. Error = ASE ("not assuming the null hypothesis"), Approx. T = z ("using the asymptotic standard error assuming the null hypothesis" – the two footnotes of the table; Pallant, 2020, chapter 16), Approx. Sig. = p. The Crosstabulation table is the agreement matrix. Note: SPSS computes κ only when both variables use the same categories; if one coder never used a certain category, the table is not square and SPSS will not print kappa – the page computes it over the full set of shared categories, as Cohen defined it.
Weighted κ: from version 27 – Analyze › Scale › Weighted Kappa…, with linear (the default) or quadratic weights and an asymptotic confidence interval; the syntax – via the dialog's Paste button. Fleiss' κ: not available in SPSS; see R below.
Krippendorff's α: there is no built-in command. The KALPHA macro by Hayes and Krippendorff (2007) – run the macro file once, and then:
KALPHA judges = coder_A coder_B coder_C /level = 1 /detail = 0 /boot = 2000.
level: 1 nominal, 2 ordinal, 3 interval, 4 ratio; boot – the number of bootstrap samples for the confidence interval (0 – none); the KALPHA algorithm samples pairs of values rather than units, so its intervals are close to those on the page but not identical. The "Download SPSS syntax" button in the output generates both parts on the loaded data, with the dictionary commands if there was a legend.
And how do you do it in R?
The irr package computes both coefficients: kappa2 – Cohen's κ for a pair (complete units only, like SPSS and like the page); kripp.alpha – Krippendorff's α for any number of coders, with NA as missing, in all four metrics. The script was run on the sample data file and gave the table values (in the comments). The site rule for R: the file is saved as UTF-8, and every comment line starts with an English word; in addition – Hebrew sheet and variable names require a UTF-8 environment (on Windows: Sys.setlocale('LC_ALL', 'Hebrew') or R version 4.2 or later, where UTF-8 is the default).
# Intercoder reliability – Professor Stat. This file is saved as UTF-8; every comment line starts with an English word
library(readxl) # reading Excel
library(irr) # kappa2, kripp.alpha
# Loading: the standard template – row = coding unit, column = coder; the sheet "data"
d <- read_excel("C:/Rdata/DATA_reliability_example_eng.xlsx", sheet = "data")
# Cohen's kappa for a pair of coders (complete units only, as in SPSS)
k <- kappa2(d[, c("coder_A", "coder_B")])
k$value # κ = .653
k$subjects # N = 139
k$statistic # z = 15.14 (test of κ = 0)
# Krippendorff's alpha – all coders, missing values allowed; rows = coders, columns = units
m <- t(as.matrix(d[, c("coder_A", "coder_B", "coder_C")]))
kripp.alpha(m, method = "nominal")$value # α = .629
kripp.alpha(m, method = "ordinal")$value # α = .659
kripp.alpha(m, method = "interval")$value # α = .654
# Weighted kappa for a pair – linear ("equal") or quadratic ("squared") weights; the categories must be ordered
kappa2(d[, c("coder_A", "coder_B")], weight = "equal")$value # linear κw = .662
kappa2(d[, c("coder_A", "coder_B")], weight = "squared")$value # quadratic κw = .664
# Fleiss' kappa – three or more coders, only units coded by all of them
f <- kappam.fleiss(d[complete.cases(d[, 2:4]), c("coder_A", "coder_B", "coder_C")])
f$value # κF = .636
f$subjects # N = 134
f$statistic # z = 24.97
kappa2 also prints z and p but not the asymptotic standard error and not κmax; kripp.alpha prints α only, without a confidence interval. The numbers in the page's table – ASE, the confidence interval, κmax and the bootstrap interval of α – are obtained from SPSS or from the page.
Reporting the results
According to APA (2020, section 6.36), κ and α – coefficients that cannot exceed 1 – are written without a leading zero and with two decimals; confidence intervals according to section 6.43. The wording the page produces: "Coding reliability between the two coders was assessed on N = 139 units. Raw agreement was 72.7%; Cohen's kappa κ = .65, 95% CI [.56, .75]; Krippendorff's alpha (nominal) α = .65". Report N, the raw agreement (Krippendorff criticizes it, but it gives the reader a sense of scale), the coefficient with its confidence interval, and the metric of α – without the metric, α cannot be interpreted. Name the interpretation yardstick you chose – Landis and Koch or Krippendorff – because they do not agree.
Common mistakes
Reporting raw agreement only. 85% agreement on a variable with a dominant category can be κ = .20. Krippendorff (2019, section 11.5) explains in detail why "%-agreement" cannot be interpreted – and why it keeps appearing.
Computing reliability on the units the coders discussed. Agreement reached in discussion is consensus, not reliability. Measure before the discussion, on independent coding.
Averaging the reliabilities of different variables – see the warning in the assumptions section.
Choosing the metric by the number that comes out. α in the interval metric is almost always higher than nominal α on the same data. The metric is determined by the nature of the variable – is the difference between 1 and 5 really greater than the difference between 1 and 2 – and documented before the computation.
Citing "slight / substantial / almost perfect" as a standard. These are Landis and Koch's words for one example, accompanied by an explicit caveat. They can be used; they should not be presented as if someone had established them.
Stopping when the coefficient is low. A low coefficient is the beginning of the work, not its end: the agreement matrix shows which categories are confused, κmax shows whether one coder leans toward a certain category, and a weak pair among three shows who interprets differently. Revise the codebook, train, and measure again on a new sample.
References
American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213-220. https://doi.org/10.1037/h0026256
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman & Hall. https://doi.org/10.1007/978-1-4899-4541-9
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323-327. https://doi.org/10.1037/h0028106
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89. https://doi.org/10.1080/19312450709336664
Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). Routledge.
Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3), 321-325. https://doi.org/10.1086/266577