Understanding Deeply · The metrics laboratory

Same test. Different answers.

A fictional screen assigns a score from 1 to 99 for an invented condition. A score at or above your threshold is a positive screen. Move the controls and follow the numbers from the confusion matrix to the final metric.

Entirely fictional · Expected counts per 1,000 people · Scores are rankings, not calibrated probabilities

Try a story

1. Where does the threshold cut?

Condition presentCondition absentThreshold

Each curve is normalized within its own group. Faded bars fall below the positive threshold. Changing prevalence leaves these shapes alone.

2. Count the four possible results

Per 1,000 · expected counts

Screen positive

Screen negative

Condition
present

True positive · TP

False negative · FN

Condition
absent

False positive · FP

True negative · TN

Decimals are intentional: these are expected counts from a model, not fractions of observed people. Calculations use unrounded values.

3. Ask a precise question

Choose a metric. Its question, denominator and arithmetic appear below.

Matrix outline: teal = numerator; purple = other cells in the denominator. Some metrics combine several rates or all thresholds.

The prevalence experiment

4. Transport the same screen to two populations

Hold the threshold and both score distributions fixed. Only the condition frequency changes. “Can change” means the metric depends on prevalence; special cases can still produce equal values.


Metric	5% prevalence	50% prevalence	Pure prevalence change

The important condition: sensitivity, specificity and ROC-AUC stay fixed here because the within-group score distributions are fixed. In real populations, disease severity, case mix or measurement can change too; those metrics can then change.

5. A point is not a whole curve

The orange point is your current threshold. Moving it does not alter the underlying curves or their summary areas.

ROC curve

Precision–recall curve

ROC-AUC measures ranking across thresholds. Average precision (AP) summarizes the precision–recall curve using recall increments; it is not trapezoidal PR-AUC. Neither selects a useful operating threshold for you.

Open the mathematical bridgeCounts → ratios → Bayes’ rule → AUC and AP

Step 1 · Build the expected counts

Let p be prevalence, Se sensitivity, Sp specificity and N = 1,000. The threshold determines Se and Sp from the fixed score distributions.

Step 2 · Put the right people in the denominator

Step 3 · See the rest of the metric family

Metric	Definition	Current result

A dash means an undefined ratio, not zero. With no positive screens, PPV is undefined; F1 is 0 here because actual positives still exist. MCC is undefined if its denominator is zero. Infinite likelihood ratios are shown as ∞.

Step 4 · Read the two area summaries

ROC-AUC = P(score of a positive > score of a negative) + ½ P(tie). The model compares every pair of score bins, weighted by the two within-group distributions. Ties receive half credit.

AP = Σ (increase in recall) × precision at that score threshold. We add one distinct score level at a time, from 99 down to 1, keeping ties together. The precision–recall plot uses this step convention. A score with no information has ROC-AUC 0.5 and AP equal to prevalence.

Step 5 · Notice what these numbers cannot tell you

F1 ignores true negatives. Accuracy can look impressive when nearly everyone belongs to the same class. AUC does not establish calibration, clinical benefit or the cost of mistakes. This score is not a predicted probability; probability metrics such as Brier score and log loss require that additional interpretation and are outside this screen-ranking experiment.

Method notes & sourcesWhat stays fixed, what is fictional, and where the definitions come from

A controlled thought experiment

Scores are the integers 1–99. Within each true class, the mass at score s is proportional to exp[−½((s − μ)/18)²], normalized over those 99 scores. The means before truncation are 50 + separation/2 and 50 − separation/2. A positive screen means score ≥ threshold. These are invented distributions, not a trained or clinically validated model.

Prevalence rescales the two classes to a total expected population of 1,000. There is no sampling noise, missing reference standard or uncertainty interval. This intentionally isolates the algebra. In a study, evaluate performance on appropriate held-out data, describe the population and reference standard, and report uncertainty.