What did the missing outcomes change?
A fictional prevention program A versus usual routine B. We want the difference in the 12-month risk of a symptom flare among everyone enrolled. Hide some outcomes, then try to reconstruct the comparison.
The data you actually see
Each row contains 500 fictional people. A hatched section is unknown to the analyst. Revealing it is a simulator privilege; it never feeds the fitted methods.
Same study. Different answers.
Risk difference = A − B, in percentage points (pp). Negative favors A for this unwanted outcome.
Known only inside this simulation25% − 40% = −15 pp
The dashed mark is the −15 pp benchmark. Each dot is a method’s point estimate. Click a row to trace its calculation. One run’s error is not the same as a method’s long-run bias.
Uncertainty matters too
Multiple imputation: several plausible completions
Within each group and selected baseline category, draw a plausible flare probability, then draw outcomes for the missing people. Analyze each completed dataset and pool the estimates and their uncertainty. These are simulated possibilities, not recovered facts.
This demonstration uses a binary-outcome model, a Beta(1,1) prior in each fitted cell, and Rubin’s pooling rules. The interval shown is an illustrative large-sample normal 95% interval—not an exact small-sample interval. It includes sampling and imputation uncertainty, but cannot protect against a wrong MAR assumption. Increasing the number of imputations reduces simulation noise; it does not replace missing information.
What if the unseen outcomes differ?
Make the assumption move
Start with the observed flare risk in each fitted cell. Suppose the missing people have a risk δ percentage points higher or lower than those observed. Change A and B separately, keeping every observed record fixed.
δ = 0 assumes the missing and observed risks match within each fitted cell. Shifted risks are limited to 0–100%; the note identifies when a limit is reached.
The mathematical bridgeOpen the counts, weights, imputations and final denominators.
1 · Begin with the same target
Estimate the 12-month flare risk among all 1,000 people in A and all 1,000 in B. Half of each group starts in each recorded risk category. The complete cohort has 250 flares in A and 400 in B: 250/1,000 − 400/1,000 = −0.15.
2 · Separate observed outcomes from missing outcomes
| Group / baseline | Total | Recorded flare | Recorded no flare | Missing | Observed risk |
|---|
3 · Complete cases change the denominator
“Do nothing” usually still makes a choice: software may drop missing rows. Counting unknown outcomes as zero is a different choice. Replacing each missing outcome by its group’s observed mean leaves the complete-case point estimate unchanged; treating those replacements as real measurements understates uncertainty.
4 · Conditional imputation and weighting reconnect the cells
Conditional single imputation adds missing count × observed risk within the fitted cell. For a binary outcome, this is an expected event count; fractional predictions are not actual patient outcomes. Response weighting gives each observed person weight 1/P(outcome observed | group, baseline). Here the response probability is estimated as observed/total in each fitted cell.
Weighted risk = Σ(weight × recorded flares) / Σ(weight × observed count)
These two point estimates coincide in this saturated, cell-based example. That is an algebraic consequence of using the same cells; different models need not agree. If a fitted cell has no observed outcomes, the app reports the model as unavailable rather than inventing evidence. Large weights reduce the effective sample size, ESS = (Σw)²/Σw².
5 · Multiple imputation adds uncertainty, not certainty
Missing flaresₛ ~ Binomial(missing countₛ, pₛ)
For completion j: Qⱼ = risk Aⱼ − risk Bⱼ
Uⱼ = risk Aⱼ(1 − risk Aⱼ)/1,000 + risk Bⱼ(1 − risk Bⱼ)/1,000
Q̄ = ΣQⱼ/M · Ū = ΣUⱼ/M
B = Σ(Qⱼ − Q̄)²/(M − 1)
T = Ū + (1 + 1/M)B · pooled SE = √T
The weak prior and finite Monte Carlo draws can make the MI point estimate differ slightly from conditional single imputation. All methods use group membership. The baseline switch chooses whether the two risk categories are modeled separately.
6 · Sensitivity analysis changes the missing-outcome model
The risk shifts are assumptions supplied by you. They are not estimated from the observed records. This simple pattern-mixture calculation shifts expected missing risks on the probability scale and reports point estimates only; a full analysis would propagate uncertainty under each sensitivity scenario.
7 · How the simulator hides outcomes
MCAR uses the same missingness probability for everyone. MAR uses logit P(missing) = α(group) + strength × high-baseline-risk. MNAR adds strength × flare-status, so dependence on the unseen outcome remains even after conditioning on baseline and group. Intercepts are calibrated using the simulator’s complete cohort to achieve the requested average missingness. Independent seeded random draws determine which outcomes disappear. That calibration and the hidden outcomes are never supplied to the analysis functions.
What these labels do—and do not—tell youAssumptions, study design and the difference between missing data and intercurrent events.
MCAR: unrelated to observed or unseen values
A purely random file loss could fit MCAR. Complete-case estimates can remain unbiased for this comparison across repeated samples, but use less information. One draw can still differ from the full-data result. With zero missingness there is nothing to repair.
MAR: explainable by information that is observed
In this example, the recorded baseline category and group account for who is missing. Within those cells, the unseen outcome adds no further selection. Ignoring the baseline variable can break the justification for an otherwise suitable method. MAR does not mean that the observed subset has the same risk mix as the original cohort.
MNAR: unseen values still matter
People with a flare might be more likely to have an unrecorded outcome even within the same baseline category. Ordinary MAR imputation and response weighting do not automatically remove this selection. MNAR analyses are possible, but need additional assumptions, information or data. The observed dataset alone generally cannot establish MAR instead of MNAR.
A method is not a label-matching exercise
Complete-case analysis can be valid beyond MCAR for some estimands and models. MAR procedures still need appropriate models, relevant predictors and adequate observations across the population. More sophisticated methods are not guaranteed to be closer in every random dataset. Preventing missingness and recording reasons and useful predictors remain valuable.
Keep the question and the missingness separate
This toy study assumes randomized groups, comparable measurement, no crossover and a meaningful 12-month flare outcome for everyone. An intercurrent event can change the meaning of the research question even when data are complete. An outcome that does not exist after death is not simply an empty cell to fill. See the intercurrent-events lesson.
Sources and model notes
- Sterne et al. (2009), Multiple imputation for missing data: potential and pitfalls — mechanisms, imputation and assumptions.
- Seaman et al. (2012), Combining multiple imputation and inverse-probability weighting — model requirements, weights and pooling.
- Leurent et al. (2018), Sensitivity analysis for not-at-random missing data — the principle of exploring departures from MAR; this app uses its own binary-outcome illustration.
- ICH E9(R1), Estimands and sensitivity analysis — the target question, missing data and sensitivity analysis.
Original educational simulation. All people, outcomes and probabilities are invented. No actual study or product is represented. Counts and the missing-data generator are deliberately simple; this dashboard is not an analysis engine for real datasets.