Errors, power & p-values.

Begin with a null hypothesis and a specified effect

Two fictional learning routines produce a continuous score. The null hypothesis says their population mean scores are equal. The simulator also knows a true difference, δ, that you control. An actual researcher would not know that value.

The demonstration uses independent normal outcomes, equal group sizes and a known common population standard deviation. Those assumptions make the two-sided z test transparent. A real analysis with an estimated standard deviation or a different design would usually need a different procedure.

Type I error, Type II error and power describe repeated decisions

A Type I error occurs when the test rejects a true null. Alpha, α, is the probability of that error under the specified test assumptions. A Type II error occurs when the test does not reject at a particular nonzero effect. Beta, β, is its probability; power is 1 − β for that effect and design.

These probabilities condition on different truths. A false-positive rate of 5% does not mean that 5% of significant findings are false. The four-cell decision table makes the conditioning explicit. Running 1,000 studies under each truth turns the theoretical probabilities into simulated frequencies that fluctuate from one run to the next.

Sample size changes precision and power

For two equal independent groups with known SD σ, the standard error of the mean difference is σ√(2/n). More people narrow the sampling distribution. More outcome variability widens it. A larger true effect places more of the alternative distribution beyond the rejection boundary.

The sliders show these changes directly, and the sample-size button finds the minimum integer group size for a chosen power target under this model. At δ = 0, the rejection probability is alpha; there is no false-null scenario for defining a Type II error. Power always needs a specified alternative effect.

A p-value belongs to the result you observed

The two-sided p-value is the probability, under the null model and its assumptions, of a mean difference at least as far from zero as the observed one. The app calculates it from the observed gap and its standard error. It does not use the simulator’s true δ.

Try changing δ while holding the observed gap, sample size and SD fixed. Power changes; the p-value does not. Then change alpha: the decision boundary changes, but the same observed result retains the same p-value. This distinction is central to the American Statistical Association’s guidance on significance and p-values.

Report the effect and its uncertainty

A p-value is not the probability that the null is true, and a non-significant result does not establish equivalence. The playground pairs every observed estimate with a confidence interval whose level is 1 − α, matching the two-sided test.

The “tiny, precise effect” example shows why statistical significance and practical importance are separate. For illustration, the page asks whether a benefit reaches 2 score points. A narrow interval can exclude zero while remaining entirely below that practical threshold.

Repeated opportunities change the false-positive risk

If all nulls are true and tests are independent, the chance of at least one false positive across k tests is 1 − (1 − α)ᵏ. At 5% for each of 20 tests, that is about 64%. The multiplicity panel compares this with a Bonferroni threshold of α/k. It also states the independence assumption behind the exact calculation.

The model equations and numerical results are available in the expandable mathematical bridge. The NIST/SEMATECH Handbook provides background on sample size, alpha, beta and detectable effects. To see why a precise result can still be systematically distorted, explore the companion real-world data bias experiments.