Post-Hoc Tests (Bonferroni, Holm, Tukey)
Info sheet · Statistics for Psychology & Neuroscience
Warning✎ Editing notes — to do / to check
Working notes for the author — not shown to students once collapsed; remove before publishing.
- [ ]
What you’ll get from this sheet
A significant ANOVA says some groups differ — post‑hoc tests say which. By the end you should be able to:
- Explain why many pairwise comparisons inflate family‑wise error.
- Apply and compare Bonferroni, Holm–Bonferroni, and Tukey’s HSD.
An ANOVA is an omnibus test — a significant F doesn’t say which groups differ. Follow up with pairwise comparisons, but correct for the many tests: Bonferroni (α ÷ number of tests), Holm (a less‑conservative step‑down), or Tukey’s HSD (built for all‑pairs). Only run post‑hocs if the ANOVA was significant.
The multiple-comparisons problem
Your decade‑of‑dance ANOVA is significant: decade affects danceability. But which decades differ — the 1990s vs the 2000s? The omnibus F won’t say, so you run post‑hoc (“after the event”) tests — all the pairwise comparisons, e.g. with pairwise.t.test. The catch: with 7 decades there are 21 pairwise comparisons, and the chance of at least one false positive balloons to about 66%. (In fMRI, where the brain is split into millions of voxels, uncorrected testing once “found” activation in a dead salmon.) So we correct.
Compare the corrections
Six pairwise comparisons with these raw p‑values. Switch the correction method and see which survive at α = .05:
Uncorrected, five of six look significant — but at a 26% family‑wise error rate, some are likely flukes. Bonferroni divides α by the number of tests (here .05 / 6 ≈ .0083), a blunt but safe cut that keeps only the strongest. Holm is smarter: it ranks the p‑values and applies the strict threshold only to the smallest, relaxing it as it steps down — so it catches a bit more while still controlling error, which is why it’s increasingly the default. Tukey’s HSD (not shown here) is purpose‑built for all pairwise comparisons, using the studentised‑range distribution; it lands between the two in strictness but must be run on the full set of pairs.
See it in code
from statsmodels.stats.multicomp import pairwise_tukeyhsd
pairwise_tukeyhsd(dance, decade) # Tukey's HSD
from statsmodels.stats.multitest import multipletests
multipletests(pvals, method="holm") # Holm correction on raw p-values[~,~,stats] = anova1(dance, decade);
multcompare(stats, 'CType', 'bonferroni') % or 'hsd' for TukeyThe R tab runs live; Python and MATLAB are static references.
TipCheck your understanding
Bonferroni and Holm both control family‑wise error, so why prefer Holm?
Because Holm is less conservative — it controls the same error rate but rejects at least as many hypotheses, so it has more power. Bonferroni applies the single strict threshold α/m to every comparison; Holm sorts the p‑values and applies α/m only to the smallest, then α/(m−1) to the next, and so on (a step‑down). You lose nothing in error control and gain sensitivity, which is why Holm has largely superseded plain Bonferroni.
Only run post‑hocs if the omnibus ANOVA was significant — trawling otherwise is “fishing” and invalid. Cut down the number of comparisons where you can (e.g. compare each condition to a single control rather than all‑vs‑all) — but note Tukey’s HSD requires the full set of pairwise comparisons, and it’s more sensitive to unequal variances than Bonferroni/Holm. Best of all, if you had specific hypotheses in advance, use planned contrasts instead — they’re more powerful and usually need no correction.
Where this shows up next
Post‑hocs are the reactive route; planned contrasts are the proactive one. See Chapter (ANOVA), and the Family‑Wise Error sheet for the inflation problem in depth.