What a p-value Actually Is (and Isn’t)
Info sheet · Statistics for Psychology & Neuroscience
Warning✎ Editing notes — to do / to check
Working notes for the author — not shown to students once collapsed; remove before publishing.
- [ ]
What you’ll get from this sheet
The most misunderstood number in science. By the end you should be able to:
- State what NHST does and the correct interpretation of a p‑value.
- Say why we never “accept” or “prove” a hypothesis.
A p‑value is the probability of data at least this extreme if the null hypothesis were true — not the probability the null is true, and not the probability your result was a fluke. In NHST we only ever reject or fail to reject the null (usually at α = .05); we never accept or prove anything.
The courtroom analogy
Think of 12 Angry Men (1957, 100% on Rotten Tomatoes). Eleven jurors vote guilty; Henry Fonda’s lone dissenter never claims the boy is innocent — only that guilt isn’t certain. By the end, the verdict is not guilty — which is not the same as “innocent.” A court can convict (guilty) or fail to convict (not guilty); it can never prove innocence.
Null‑hypothesis significance testing (NHST) works the same way. We start by assuming there is no effect — no difference, no relationship — the null hypothesis. We then look for evidence against it. If the evidence is insufficient, we fail to reject the null (which does not prove it true). If the evidence is strong enough, we reject it and infer there’s an effect (which we still haven’t proven). All we can ever say is that the probability of observing data like ours by chance, if the null were true, is below some threshold — usually 0.05.
The five steps: (1) state the research question; (2) identify the null and alternative hypotheses; (3) set a significance level α (commonly .05); (4) collect data and compute a test statistic and its p‑value; (5) compare — if p < α, reject the null; if p ≥ α, fail to reject. Note the careful wording throughout: we never “accept”, “prove”, “confirm”, or “demonstrate” — only reject or fail to reject.
The p-value is a tail area
Under the null, your test statistic has a known distribution. The p‑value is the area in the tail(s) beyond your observed statistic — how much of the “if the null were true” world is at least as extreme as what you saw. Slide the observed statistic and watch the tail area (the p‑value) shrink:
The blue curve is the world if the null were true. Your observed statistic sits somewhere along it, and the shaded tails are everything at least as extreme — that area is the p‑value. Push the statistic out toward the tails and the area shrinks below .05 (past the ±1.96 markers), and you reject. The crucial reading: that shaded area is P(data this extreme | null true) — it is not the probability the null is true, and not the probability your finding was a fluke.
The limitation NHST can’t escape
Notice the asymmetry: failing to reject the null teaches you almost nothing. It doesn’t mean there’s no effect — only that this study didn’t find enough evidence. Absence of evidence isn’t evidence of absence. This is exactly where Bayesian inference becomes appealing: instead of a binary reject / fail‑to‑reject, a Bayes factor quantifies the relative evidence for both hypotheses (a Bayes factor of 3 means the data are 3× more likely under one hypothesis than the other; below 1 it favours the null). So you learn something useful whichever way the result falls — a theme the Bayesian sheets pick up.
See it in code
TipCheck your understanding
Your test gives p = 0.03. Which of these is the correct interpretation? (a) There’s a 3% chance the null is true. (b) There’s a 97% chance your effect is real. (c) If the null were true, you’d see data this extreme only 3% of the time.
(c). A p‑value is computed assuming the null is true — it’s the probability of data at least this extreme under H₀. It says nothing directly about the probability that H₀ is true (a) or that your effect is real (b); those require a Bayesian framing. p = .03 lets you reject the null at α = .05, but “reject” is a decision, not a proof.
The four classic errors, worth vaccinating against: a p‑value is not the probability the null is true, nor the probability your result was “due to chance.” A non‑significant result is not proof of no effect (absence of evidence ≠ evidence of absence). “Statistically significant” ≠ “large” or “important” — with a big enough sample a trivial effect clears the bar. And a small p on a badly designed study is precise nonsense: NHST can’t rescue a confounded design.
Where this shows up next
The t‑tests, ANOVAs, and regressions throughout the book all end in a p‑value read exactly this way. The Bayesian sheets offer the alternative that can evidence the null. See Chapter (Inference).