It All Comes Down to Noise
Info sheet · Statistics for Psychology & Neuroscience
Warning✎ Editing notes — to do / to check
Working notes for the author — not shown to students once collapsed; remove before publishing.
- [ ]
What you’ll get from this sheet
The single idea the rest of the book is built on. By the end you should be able to:
- Say why variance is the subject, not a nuisance.
- Explain why significance is always signal relative to noise.
Almost everything in statistics is an argument about noise. Every test — from a t‑test to a mixed model — compares the variation it can explain (signal) against the variation it can’t (noise). “Significant” means the signal is large relative to the noise.
Variance is the subject
If you take one idea from this book, let it be this: almost everything in statistics is an argument about noise. Measure the same thing twice — one person’s reaction times on two days, a neuron’s firing rate across identical trials, two classes taught the same way — and you never get the same numbers. The values wobble. Some of that wobble is interesting (a real effect of caffeine on reaction time); some is noise — the countless tiny, uncontrolled influences (attention, temperature, what they had for breakfast) that push a measurement around for reasons unrelated to your question.
That wobble has a name you know: variance. Variance isn’t a nuisance we tolerate on the way to the “real” statistics — variance is the subject. Every test in this book is a machine for one thing: comparing the variation we can explain against the variation we cannot. When people say a result is “significant,” they mean the signal is large relative to the noise. A big effect drowning in noise can be undetectable; a tiny effect measured with exquisite precision can be unmistakable. Signal alone tells you almost nothing — it’s always signal per unit of noise.
Same effect, different noise
Two groups whose true means differ by a fixed amount. The signal (that gap) never changes — only the noise does. Slide it and watch whether the difference is visible or buried:
The gap between the two means is fixed at 3 the whole time — yet at low noise the groups are obviously different, and at high noise they melt into one blur. Nothing about the effect changed; only the noise did. That ratio, signal ÷ noise, is what every test is really measuring — here as Cohen’s d, elsewhere as an F‑ratio, a t‑statistic, or an R².
Keep the lens close
This is why, before running a single test, we spend so long thinking about where variance comes from and how to measure it. As each new method appears, ask: what variance is this technique trying to explain, and what variance is it treating as noise? The whole edifice of inferential statistics is built from that one question.
See it in code
TipCheck your understanding
Two studies find the exact same difference in means between conditions, but one is “significant” and the other isn’t. How is that possible?
Because significance depends on the difference relative to the noise, not the difference alone. The significant study measured with less noise — smaller within‑group variance and/or a larger sample (which shrinks the standard error). Same signal, less noise → bigger signal‑to‑noise ratio → detectable. It’s never the size of the effect on its own; it’s the effect per unit of noise.
Don’t read “not significant” as “no effect” — a real effect can be swamped by noise (low power), and “significant” as “big / important” — a trivial effect can clear the bar with a huge, precise sample. Report an effect size (like Cohen’s d or R²) alongside the p‑value so the signal and the noise are both visible.
Where this shows up next
This lens returns everywhere. Next, Signal and Error makes it an equation (Data = Signal + Error); the F‑ratio, R², and t‑statistic are all just versions of the same ratio.