Cohen’s kappa calculator for inter-rater agreement

Cohen (1960) proposed kappa for two raters who each assign every subject to one of the same categories. Raw percent agreement overstates reliability because two raters who guess will still agree some of the time. Kappa subtracts that expected agreement: pₑ is the sum, over categories, of the product of the two raters’ marginal proportions. In the table above rater A says "yes" 45% of the time and rater B 50% of the time, so the chance both say yes is 0.45 × 0.50 = 0.225; both say no is 0.55 × 0.50 = 0.275; and pₑ = 0.50. Kappa is 1 for perfect agreement, 0 when agreement is no better than chance, and can go negative when raters disagree more than chance predicts.

The large-sample standard error most online calculators use is SE(κ) = √[pₒ(1 − pₒ) / (n(1 − pₑ)²)]. Here that is √[0.85 × 0.15 / (100 × 0.25)] = 0.0714, so the 95% interval is 0.70 ± 1.96 × 0.0714, or 0.56 to 0.84. The fuller variance of Fleiss, Cohen and Everitt (1969) gives 0.0711 for this table, practically the same. With small samples, rare categories or kappa near 1, the normal approximation is poor, and a statistician may prefer a bootstrap or exact interval. The width matters as much as the point estimate: with 100 events the interval spans roughly "moderate" to "almost perfect" on the usual scale.

Landis and Koch (1977) labelled 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and above 0.80 almost perfect. Those labels were an arbitrary convention and have been criticised for being applied as pass/fail thresholds (McHugh, 2012). A kappa of 0.70 is acceptable for a screening flag that a clinician will re-check and may be far too low for a rating that decides primary endpoint status without any second look. Decide the required agreement from the consequence of a disagreement, write it into the adjudication charter or the statistical analysis plan before looking at the data, and judge the lower bound of the interval, not only the point estimate.

When categories are ordered (none, mild, moderate, severe), a disagreement between "mild" and "severe" is worse than one between "mild" and "moderate". Weighted kappa (Cohen, 1968) gives partial credit: with linear weights the credit for a disagreement is 1 minus the distance between categories divided by the maximum distance, and with quadratic weights it is 1 minus the squared relative distance. Take 70 subjects rated mild, moderate or severe by two raters with the counts 20, 5, 0 / 4, 15, 6 / 1, 3, 16 (rows rater A, columns rater B). Unweighted kappa is 0.59, linear-weighted kappa is 0.68 and quadratic-weighted kappa is 0.76. The statistic you report must be the one named in the analysis plan, with the weighting scheme stated, because the choice alone moved the result by 0.17.

Kappa is defined for two raters rating every subject. With three or more raters, Fleiss’ kappa (1971) generalises it, and if the raters differ between subjects a different statistic (Krippendorff’s alpha, for example) may fit better. For measurements on a continuous scale, such as a blood pressure read by three nurses or a tumour diameter measured by two radiologists, dichotomising to compute kappa throws away information. The intraclass correlation does the right job there, and the ICC calculator works through Shrout and Fleiss forms with a numerical example.

Plan the agreement study as carefully as the trial. The standard error formula above can be turned around: to estimate a kappa near 0.70 with chance agreement near 0.50 to within ±0.10 at 95% confidence, you need about 196 subjects rated by both raters; ±0.15 needs about 87, and ±0.05 needs about 784. Those figures assume a balanced table, and a rarer category needs more. Before the real ratings start, run a calibration set of cases, discuss the disagreements, tighten the written definition and only then collect the ratings that will be reported. Record who the raters are and their training, because a kappa from two experienced readers does not transfer to a pool of new ones. State in the charter how a disagreement is resolved (a third reviewer, a consensus meeting) and report the agreement before the tie-break, since the resolved value is by construction in perfect agreement.

Agreement statistics are only as good as the independence of the ratings. Build the rating as its own eCRF form with single-choice fields for the category and a free-text field for the reasoning, completed separately by each rater, with the independence written into the adjudication procedure. In Capture, forms follow a draft-to-approved lifecycle, so the rating form is locked once approved and both raters use the same version, and role-based access separates site, researcher and admin permissions. Every entry carries a field-level audit trail with user, timestamp, old value, new value and reason for change, which lets you check the timestamps and show later whether a rating was edited after the other rater’s call was entered.

The ratings export as CSV or Excel in wide format with a data dictionary, one row per submission, so the two raters’ category columns can be tabulated into the 2x2 table above without reshaping. Edit checks keep the data clean at entry: a required single-choice field prevents blanks, and a custom rule can raise an auto-query when a free-text justification is missing for a "yes". For safety data, the same pattern fits the adverse event reporting workflow and the AE eCRF template. Kappa itself is computed in your statistical software or the table above; Capture supplies the clean, audit-trailed inputs. For diagnostic studies where agreement between readers is itself an outcome, see EDC for diagnostic accuracy studies.