NHST vs Bayesian Inference

Info sheet · Statistics for Psychology & Neuroscience

Author

Andrew Bell

Published

August 13, 2026

Info sheet 0.2 (draft) · Prerequisites: what a p-value is (NHST) · Give feedback ↗

Working notes for the author — not shown to students once collapsed; remove before publishing.

  • [ ]

What you’ll get from this sheet

Two frameworks for weighing evidence — and the different questions they ask. By the end you should be able to:

  1. State the question NHST answers, and its blind spot.
  2. State the question Bayesian inference answers instead.

NHST asks: if the null were true, how surprising is our data? — and can only reject or fail to reject the null. Bayesian inference asks: given the data, how much more likely is one hypothesis than another? — comparing two models head‑to‑head and reporting the relative evidence as a Bayes factor. Crucially, Bayes can gather evidence for a null; NHST cannot.

The same question, two ways

Recall the 12 Angry Men courtroom analogy: the jury never declares the accused innocent, only “not guilty” for lack of evidence. That’s NHST exactly. We assume the null — no effect, nothing of interest — and ask a single question: if the null were true, how surprising would our data be? If the data are surprising enough (p < .05), we reject the null. We never accept it; a non‑significant result is “not guilty,” not “innocent.” NHST can say the data are unlikely under the null, but it cannot say how likely the null itself is, and it cannot gather evidence for a null.

The Bayesian approach asks a different — arguably more natural — question: given the data we actually observed, how much more likely is one hypothesis than another? Rather than a single yes/no verdict against a straw‑man null, it compares two models head‑to‑head and reports the relative evidence for each. Flip between the two lenses on the same borderline result:

Feed both frameworks the same borderline dataset and the contrast is stark. NHST returns p = .08, “fail to reject” — a verdict that teaches you almost nothing, because a non‑significant p can’t distinguish “there’s no effect” from “we didn’t gather enough evidence.” The Bayesian analysis returns a Bayes factor, a single number saying how the data shift the balance between the two hypotheses — here barely at all, but it can tell you the null is (weakly) favoured, and it grows as you collect more data. That ability to evidence a null is the headline advantage.

See it in code

# NHST:
t.test(group_a, group_b)$p.value        # p — reject or fail to reject only

# Bayesian (BayesFactor package):
library(BayesFactor)
ttestBF(group_a, group_b)               # BF10 — relative evidence for effect vs no effect

A study reports p = .30 and concludes “there is no effect.” What’s wrong, and how would a Bayesian analysis help?

The conclusion overreaches: p = .30 means only that the data weren’t surprising under the null — you fail to reject, which is “not guilty,” not “innocent.” NHST literally cannot provide evidence for the null. A Bayesian analysis can: a Bayes factor compares the null and alternative directly, so a BF₀₁ of, say, 6 would be genuine, quantified evidence favouring no effect — or a BF near 1 would tell you the data are simply inconclusive, which is very different from “no effect.”

Bayesian inference isn’t “NHST with nicer words” — it needs a prior (a statement of what’s plausible before the data), and the Bayes factor can depend on that choice, so priors should be reasonable and reported. And a Bayes factor is relative evidence between the two models you specified — it says nothing about a third hypothesis you didn’t test. It’s a better tool for many questions, not a magic one.

Where this shows up next

Next, how to read the number Bayes reports — the Bayes factor — and then Bayesian versions of ANOVA and regression. See Chapter (Bayesian Methods).