Performance metrics.

Begin with the denominator

A positive screen and a true condition are different things. Sensitivity starts with everyone who truly has the condition and asks how many the screen finds. Specificity starts with everyone without it and asks how many the screen correctly clears. Positive predictive value (PPV, or precision) starts with positive screens. Negative predictive value (NPV) starts with negative screens.

The playground makes those four starting points visible. Choose a metric and its numerator and denominator light up in the confusion matrix. The expected counts come from a completely invented population of 1,000; no real patient records or study results are used.

Why prevalence changes the answer

Imagine that sensitivity is 80% and specificity is 90%. At 50% prevalence, 1,000 people yield 400 true positives and 50 false positives: PPV is 400/450, or 88.9%. At 1% prevalence, the same performance yields 8 true positives and 99 false positives: PPV is 8/107, or 7.5%. Those are separate hand-calculated examples; the controls use a full score distribution.

Changing prevalence alone preserves sensitivity, specificity and ROC-AUC when the conditional score distributions and operating threshold stay fixed. PPV, NPV, F1, accuracy, average precision and MCC can change. This is a controlled assumption, not a guarantee that a model retains its performance in another real population.

A threshold and an area answer different questions

Moving the threshold changes who screens positive. It moves a point along the ROC and precision–recall curves. ROC-AUC summarizes ranking across all thresholds, with half credit for ties. Average precision summarizes the precision–recall relationship using changes in recall. A prevalence shift can change the latter even when ROC-AUC stays fixed.

F1 combines precision and recall but ignores true negatives. Accuracy can hide a failure to identify rare positives. Balanced accuracy, error rates, likelihood ratios and Youden’s J offer other views; the expanded calculations show them alongside MCC. Probability calibration and the consequences of false positives and false negatives remain separate questions.

Use the experiment to build intuition

Try the rare-condition example, then the accuracy trap. Keep one control fixed while changing another. The equations, curves and comparison table update together, so the relationship between prevalence and each metric stays inspectable. Definitions and conventions are linked in the playground’s method notes, including the metric evaluation reference.

For a different kind of comparison, explore how matching and IPTW change group outcomes, or move on to designing a target trial emulation.