One study’s p-value. Many studies’ power.
Two invented learning routines, A and B, produce a continuous task score. Higher is better. Test H₀: their population mean scores are equal. This simplified experiment uses independent normal outcomes and a known common standard deviation.
Before seeing the data
Now inspect one study
Start with an illustrative observed gap, or draw simulated group means.
Changing δ alone leaves this observed gap and its p-value unchanged. In a real study δ is unknown. Changing n or σ here explores a hypothetical study with the same observed gap.
The four possible outcomes of a test
The columns are different hypothetical truths. α and β condition on different columns; they are not the probabilities that a particular conclusion is wrong.
| Test decision | H₀ is true: δ = 0 | |
|---|---|---|
| Reject H₀ | ||
| Do not reject H₀ |
Run the experiment to compare theoretical probabilities with simulated frequencies.
What if you try many tests?
One pre-specified test at α = 5% is different from looking for a small p-value among many tests. This panel assumes all null hypotheses are true and the tests are independent.
The mathematical bridgeSample size → standard error → rejection boundary → power → observed p-value.
Interpretation, assumptions and sourcesA p-value is not the probability that the null is true.
What each quantity means
α is the long-run chance of rejection when H₀ is true, under the test’s assumptions. β is the chance of not rejecting at a specified nonzero effect. Power is 1 − β for that effect and design. A two-sided p-value measures how often H₀ would produce a mean difference at least as far from zero as the one observed, under the model. These quantities condition on different things.
What the result does not say
A p-value does not give the probability that H₀ is true, the probability the result occurred “by chance,” the size of an effect or its practical importance. Not rejecting H₀ does not establish no effect or equivalence. Statistical significance does not repair confounding or measurement bias. See the bias experiments.
Planning and reporting
Power here is calculated for a specified effect before interpreting study results. Replacing that effect with the observed estimate to calculate “observed power” is not a substitute for reporting the estimate and confidence interval. The displayed interval has confidence level 1 − α, matching the two-sided test. In repeated use under the model, that interval procedure covers the fixed population difference at the stated rate; it is not a posterior probability for this particular interval.
A deliberately simple model
Both fictional groups have independent, normally distributed scores and the same known population SD. B’s population mean is 50 and A’s is 50 + δ. The app draws sample means directly from their exact normal sampling distributions. It does not manufacture individual records. In real analyses, an estimated SD generally calls for a suitable t procedure or another model, and clustering, missingness and study design may require different methods.
Multiple testing
For independent true nulls, the chance of at least one false positive is 1 − (1 − α)ᵏ. Bonferroni uses α/k per test to control the family-wise error at at most α, including with dependent tests. It usually reduces power for each individual test. This toy panel does not model repeated peeking or optional stopping; those also require appropriate design and analysis.
- American Statistical Association: statement on significance and p-values.
- NIST/SEMATECH Handbook: sample size, α, β and detectable effects. This app extends the known-SD normal calculation to two equal independent groups.
- NIST/SEMATECH Handbook: Bonferroni multiple comparisons.
All scores and settings are invented for teaching. Numerical normal probabilities use an approximation; the app labels very small probabilities as bounds. No real study or product is represented.