Rater agreement.

Begin with the same cases and independent judgments

Imagine two physicians reviewing 100 invented case descriptions. Each marks a feature as present or absent. There is no known correct answer in this exercise: the aim is to understand whether the physicians agree. The four cells of the paired table distinguish both-present, both-absent and the two directions of disagreement.

That is a different question from accuracy. A pair of physicians could agree on every case while both misunderstand the task. Treating one physician as a gold standard would introduce another assumption. The companion performance metrics lesson explores comparisons against a known outcome.

Cohen’s kappa starts with a reference model

The default fictional table has 45 both-present ratings, 45 both-absent ratings and 10 disagreements. Observed agreement is 90%. Each physician uses each category half the time, so independent ratings with those category proportions would agree 50% of the time.

Cohen’s calculation is κ = (0.90 − 0.50) / (1 − 0.50) = 0.80. The numerator measures agreement above the marginal benchmark; the denominator measures the remaining room above that benchmark. The original paper by Cohen introduced this approach for nominal categories. The word “chance” refers to the specified reference model, not the probability that a physician guessed.

Keep 90% agreement and make positive ratings rare

Move the slider until nobody is rated present by both physicians. There are now 90 both-absent cases and five disagreements in each direction. Observed agreement remains 90%, but each physician marks only 5% present. Their marginal benchmark becomes 0.05² + 0.95² = 0.905, producing κ ≈ −0.053.

That seemingly surprising result comes directly from the denominator and reference model. The category-specific summaries add context: present agreement is zero, while absent agreement is about 94.7%. The expandable AC1 panel introduces a different reference term. Gwet’s AC1 can be informative, but it is not interchangeable with Cohen’s kappa. Choosing the larger number is not a defensible analysis strategy.

Inter-rater and intra-rater describe who repeats the judgment

Inter-rater agreement compares different people assessing the same material. Intra-rater agreement compares the same person’s assessments on separate occasions. For two categorical assessments, either design can produce a paired table for Cohen’s kappa. Switching the design in the playground deliberately retains the numbers: it changes the question, not the arithmetic.

A repeat assessment should use stable material and protect against recall of the earlier answers. A physician can be internally repeatable yet disagree with colleagues. If the underlying condition genuinely changes between assessments, a disagreement may reflect that change rather than inconsistent rating.

Fleiss’ kappa averages agreement within a panel

The next station lets you edit ratings from two to six physicians on 12 fictional cases. Four ratings create six distinct pairs per case. If three physicians say present and one says absent, three of those six pairs agree: the case-level agreement is 50%, even though the panel is not unanimous.

Fleiss’ method averages those case-level fractions and uses pooled category proportions for its reference term. It treats ratings as exchangeable and, in this implementation, requires equal numbers of ratings per case. It is not an average of pairwise Cohen kappas. Select two ratings per case to see that Cohen and Fleiss can differ even on the same data because their reference models differ.

Ordered grades deserve an explicit disagreement rule

With low, medium and high grades, low versus medium may be less consequential than low versus high. Weighted kappa lets the analyst specify partial agreement credit. In the lesson, adjacent categories receive 0.5 under linear weights or 0.75 under quadratic weights; opposite extremes receive zero.

The same weights must enter both observed and expected agreement. The side-by-side results show why the choice matters. A larger weighted kappa does not mean anyone changed their mind or became more accurate. The weighting rule changed the summary.

Numeric scores need a different lens

In the continuous-score experiment, add 12 points to every score from physician B while removing noise. Correlation remains perfect. So does consistency ICC, which allows an additive rater offset. Absolute-agreement ICC responds to the numeric mismatch. The lesson names its specification: two-way mixed effects, single measurement, limited to the two physicians. Koo and Li explain why reporting the ICC model, type and definition matters.

The Bland–Altman difference plot returns to the original units. Its mean difference and estimated limits show how far individual measurements can separate. Those limits are not a confidence interval for the average difference. The tolerance slider is an illustrative judgment; real acceptability criteria require justification.

Report the design and distribution, not just a coefficient

Each station connects its inputs to an expandable mathematical bridge. All outputs are teaching point estimates from invented data. A real report also needs uncertainty, a justified case mix, the rating protocol and an analysis suited to missingness or clustered observations. There is no universal kappa threshold that establishes safe clinical use, and “intraclass” in ICC does not mean “intra-rater.”