Family-Wise Error
Info sheet · Statistics for Psychology & Neuroscience
Warning✎ Editing notes — to do / to check
Working notes for the author — not shown to students once collapsed; remove before publishing.
- [ ]
What you’ll get from this sheet
By the end you should be able to:
- Explain why running many tests on the same data inflates false positives.
- Count the pairwise comparisons among k groups and describe how that number grows.
- Compute the family-wise error rate — and play with it interactively below.
The more tests you run on the same data, the more likely at least one “significant” result is a false alarm. You have to budget for that risk before interpreting any single test.
The problem in one sentence
Every significance test carries a small risk of a false positive (a Type I error). Run enough tests and that small per-test risk compounds into a large chance that at least one “significant” result is a fluke. That accumulated risk is the family-wise error rate (FWE).
How many comparisons?
If you want to compare every pair of k groups, the number of comparisons is
\[ \binom{k}{2} = \frac{k(k-1)}{2}. \]
It’s worth being precise here: this grows quadratically with k (like \(k^2\)), not exponentially.
| Groups (k) | Pairwise comparisons |
|---|---|
| 2 | 1 |
| 3 | 3 |
| 4 | 6 |
| 5 | 10 |
| 6 | 15 |
The family-wise error rate
If you run m independent tests, each at significance level \(\alpha\), the chance of at least one false positive is
\[ \text{FWE} = 1 - (1-\alpha)^m. \]
A quick intuition pump: rolling a fair die, the chance of getting at least one six in 10 rolls is \(1 - (5/6)^{10} \approx 0.838\) — about 84%. Each roll is unlikely to be a six, but across ten rolls a six becomes very likely. Type I errors behave the same way across many tests.
Try it yourself
Change alpha and m below and press Run. What value of m pushes the family-wise error past 50%?
alpha = 0.05; % per-test significance level
m = 10; % number of independent tests
fwe = 1 - (1 - alpha)^mReference code — identical in MATLAB and Octave. Browser cells can’t execute MATLAB/Octave, so this tab is static (copy it into MATLAB or free Octave to run it).
And here’s the whole curve — the reason “just run all the t-tests” is dangerous:
m = 1:20;
plot(m, 1 - (1 - 0.05).^m, 'o-'); hold on
plot([1 20], [0.05 0.05], '--') % the nominal per-test rate
xlabel('Number of comparisons')
ylabel('Family-wise error rate')
title('P(at least one false positive)')Static reference (MATLAB/Octave). The R and Python tabs run live in the page.
TipCheck your understanding
With 4 groups, how many pairwise comparisons are there, and what is the family-wise error rate at α = .05?
4 groups → \(4\times3/2 = \mathbf{6}\) comparisons. FWE \(= 1 - 0.95^{6} \approx \mathbf{0.26}\) — about a 26% chance of at least one false positive. (Confirm it by setting m <- 6 in the cell above.)
See it: false positives across a “brain”
fMRI shows the problem at its most dramatic. A single scan carves the brain into tens of thousands of voxels, and the standard analysis runs a separate statistical test at every one. At an uncorrected α = .05, roughly 1 in 20 voxels crosses the threshold by chance alone — thousands of phantom “activations.” The famous demonstration is Bennett and colleagues’ dead Atlantic salmon, which appeared to show brain activity in response to emotional photographs when the authors deliberately skipped multiple‑comparisons correction.
The toy axial slice below hides one real activation (it shows up in green) in a field of otherwise pure noise (red = false positive). Drag the threshold, then switch the correction method, and watch what each approach keeps versus throws away:
With no correction, the single genuine cluster (green) is hard to pick out among the scattered red false positives. Bonferroni is safe but blunt — strict enough to erase a weak signal along with the noise. Cluster‑extent correction exploits the fact that true activations are spatially contiguous while noise is scattered: it keeps the real cluster and clears the rest — essentially how fMRI turns a field of noise into a trustworthy map.
What to do about it
The fix is to control the family-wise error — spend your α budget carefully across the comparisons. The most common tools are the Bonferroni and Holm–Bonferroni corrections and Tukey’s HSD, and where a single omnibus question suffices, an ANOVA avoids the problem entirely.
These corrections are covered in full in Chapter 8 (Analysis of Variance) of the book — this sheet is the stand-alone primer. ## Related topics
- What a p-value Actually Is — what the threshold you’re correcting actually means.
- Post-Hoc Tests — Bonferroni, Holm and Tukey corrections in practice.
- Planned Contrasts — testing specific hypotheses without a scattergun of tests.
- Why Not Just Run Lots of One-Way ANOVAs? — another cost of piling up tests.