Same case. Same judgment?
Two fictional physicians review the same invented case descriptions independently. They mark a feature as present or absent. Change their ratings and watch agreement become a statistic. Neither physician is a gold standard.
Build the 2 × 2 table
100 casesEdit a count. Each case contributes to exactly one cell.
| First rating | B: present | B: absent | Total |
|---|---|---|---|
| A: present | |||
| A: absent | |||
| Total |
Shaded cells are agreements. Off-diagonal cells are disagreements, not known mistakes.
Where do they agree?
Category-specific agreement is symmetric: neither rating is treated as truth. These are not sensitivity and specificity.
Another chance model: Gwet’s AC1Why a different coefficient can give a different answer.
Tap a rating. Change the panel.
Every case gets the same number of ratings. Buttons toggle between present (+) and absent (−). Fleiss’ calculation counts the agreeing pairs within each case; it does not require unanimity.
Which chance benchmark?
What happened to physician identity?
Fleiss’ kappa pools category use across ratings. This illustration uses a fixed panel, but the statistic treats ratings as exchangeable; it cannot reveal which physician is consistently stricter. The classical design permits different raters across cases, sampled from a common rater population, with equal ratings per case. A fixed panel with distinct rating habits may call for a method retaining rater-specific marginals, such as Conger’s generalized kappa.
Low, medium or high?
Two physicians assign an ordered grade to each fictional case. Edit the counts or use the slider. Neither grade is a known truth.
The slider also changes category marginals, so the expected weighted agreement changes too. This is not a change in disagreement penalties alone.
Moving together ≠ giving the same value
24 invented cases receive a continuous score in arbitrary points from the same two physicians. Compare a single physician’s measurement, not an average of both.
ICC specification: two-way mixed effects, single measurement, with either absolute agreement, ICC(A,1), or consistency, ICC(C,1). Conclusions concern these two physicians. “Intraclass” does not mean “intra-rater.”
Dashed lines: estimated 95% limits of agreement, mean difference ± 1.96 SD of differences. These describe individual differences under a normal, constant-variance model; they are not a confidence interval for the mean. Shading shows your illustrative tolerance.
The 24 fictional score pairsInspect the values used in the ICC and difference plot.
The mathematical bridgeCurrent ratings → observed agreement → reference model → final result.
Inter-rater and intra-rater describe the design
Inter-rater: between people
Physician A and physician B assess the same material independently. The question is whether their judgments agree. For two categorical ratings, Cohen’s kappa can summarize the paired table.
Intra-rater: within one person
Physician A rates the same stable material twice, without seeing the earlier answer. Cohen’s kappa can use those two occasions as the paired ratings. An appropriate interval and reordered cases help reduce recall. Real change in the material must not be mistaken for poor repeatability.
A person can be very repeatable while disagreeing with another person. Both can also agree and both be wrong. Agreement, repeatability and accuracy are different questions.
Choosing a measure, limitations and sourcesNo universal “good kappa” cutoff, and no gold standard in these examples.
| Data and question | Useful starting point | What to state |
|---|---|---|
| Two categorical ratings | Raw agreement, Cohen’s kappa, category-specific agreement | Paired table and each rater’s category proportions |
| Several categorical ratings per case | Fleiss’ kappa for a pooled, exchangeable-rater benchmark | How raters were selected; equal ratings per case here |
| Ordered categories, two ratings | Weighted Cohen’s kappa | Order and pre-specified weight matrix |
| Highly unequal category use | Inspect raw and specific agreement; AC1 can add another perspective | Its different reference model; avoid choosing the biggest coefficient |
| Continuous measurements | ICC plus differences and a Bland–Altman plot | ICC model, single or average measurement, consistency or absolute agreement |
| Missing or unequal numbers of ratings | Consider an appropriate Krippendorff’s alpha or a model for the actual design | Measurement scale, missingness and clustering; do not fill missing ratings as “absent” |
All outputs here are descriptive point estimates from invented data, without inferential confidence intervals. Real studies need a planned case mix, independent assessments, adequate sample size, uncertainty estimates and handling of clustered cases. Kappa depends on category use; ICC depends on between-case variability. A coefficient by itself does not establish clinical acceptability. Negative estimates are shown rather than forced to zero.
The continuous simulator holds a set of normal draws fixed while you move a slider. It centers and scales the latent-case and error vectors to the selected sample SDs so that the controls are exact within this teaching example. B has an added fixed offset. New draws change the pattern of errors. No real measurements are represented.
- Cohen (1960): A coefficient of agreement for nominal scales.
- Fleiss (1971): Measuring nominal scale agreement among many raters.
- Cohen (1968): Weighted kappa.
- Gwet (2008): Agreement in the presence of high agreement; Vach & Gerke (2023): Why AC1 is not interchangeable with Cohen’s kappa.
- Koo & Li (2016): Choosing and reporting an ICC.
- Bland & Altman (1986): Assessing agreement between measurements.
- Conger (1980): Generalized kappas for multiple raters.
- Krippendorff: Alpha reliability and incomplete ratings.