NHST vs Bayesian Inference

Info sheet · Statistics for Psychology & Neuroscience

Author

Andrew Bell

Published

September 22, 2026

Info sheet 0.2 (draft) · Prerequisites: what a p-value is (NHST) · Give feedback ↗

Working notes for the author — not shown to students once collapsed; remove before publishing.

What you’ll get from this sheet

Two frameworks for weighing evidence — and the different questions they ask. By the end you should be able to:

  1. State the question NHST answers, and its blind spot.
  2. State the question Bayesian inference answers instead.

NHST asks: if the null were true, how surprising is our data? — and can only reject or fail to reject the null. Bayesian inference asks: given the data, how much more likely is one hypothesis than another? — comparing two models head‑to‑head and reporting the relative evidence as a Bayes factor. Crucially, Bayes can gather evidence for a null; NHST cannot.

The same question, two ways

You ever see the movie 12 Angry Men? It’s an absolute banger of a film. The entire film takes place in a jury room where 12 men debate the guilt of an unseen defendant. Henry Fonda kicks off the movie’s central premise by choosing to vote ‘not guilty’ vs. the 11 votes for guilty. When asked why, he states “I’m not saying the defendant is innocent, I’m just saying that it’s possible”. I won’t spoil the film except to say that by the end of the film, the entire jury finds the defendant not guilty (Oops… I guess I did spoil the film. Sorry).

But here’s the thing. They did NOT declare the accused innocent but rather not guilty. That is NHST in a nutshell.

We assume the null — no effect, nothing of interest — and ask a single question: if the null were true, how surprising would our data be? If the data are surprising enough (p < .05), we reject the null. We never accept it; a non‑significant result is “not guilty,” not “innocent”. NHST can say the data are unlikely under the null, but it cannot say how likely the null itself is, and it cannot gather evidence for a null.

The Bayesian approach asks a different — arguably more natural — question: given the data we actually observed, how much more likely is one hypothesis than another? Rather than a single yes/no verdict against a straw‑man null, it compares two models head‑to‑head and reports the relative evidence for each.

If our court system worked using a Bayesian approach, we might find ourselves saying “The defendant is 15 times more likely to be innocent compared to guilty”.

Flip between the two lenses on the same borderline result:

Feed both frameworks the same borderline dataset and the contrast is stark. NHST returns p = .08, “fail to reject” — a verdict that teaches you almost nothing, because a non‑significant p can’t distinguish “there’s no effect” from “we didn’t gather enough evidence.” The Bayesian analysis returns a Bayes factor, a single number saying how the data shift the balance between the two hypotheses — here barely at all, but it can tell you the null is (weakly) favoured, and it grows as you collect more data. That ability to evidence a null is the headline advantage.

See it in code

# NHST:
t.test(group_a, group_b)$p.value        # p — reject or fail to reject only

# Bayesian (BayesFactor package):
library(BayesFactor)
ttestBF(group_a, group_b)               # BF10 — relative evidence for effect vs no effect
from scipy import stats
import pingouin as pg
print(stats.ttest_ind(group_a, group_b).pvalue)   # NHST p-value
print(pg.ttest(group_a, group_b)["BF10"])         # Bayesian: BF10 (pingouin)
[~, p] = ttest2(group_a, group_b);   % NHST p-value
% MATLAB has no built-in Bayes factor. Approximate via BIC —
%   BF10 ≈ exp((BIC_null − BIC_alt)/2) — or use the 'bayesFactor' toolbox (File Exchange).

All three are static references — BayesFactor / pingouin aren’t in the in-page runtime; run them locally.

A study reports p = .30 and concludes “there is no effect.” What’s wrong, and how would a Bayesian analysis help?

The conclusion overreaches: p = .30 means only that the data weren’t surprising under the null — you fail to reject, which is “not guilty,” not “innocent.” NHST literally cannot provide evidence for the null. A Bayesian analysis can: a Bayes factor compares the null and alternative directly, so a BF₀₁ of, say, 6 would be genuine, quantified evidence favouring no effect — or a BF near 1 would tell you the data are simply inconclusive, which is very different from “no effect.”

So hopefully, you can see that while the Bayesian approach might not work effectively for our court system, it offers significant advantages (see what I did there?) over NHST - under the right circumstances.

Is Bayes the right approach always? No. But it IS a valuable tool to have at your disposal.

Bayesian inference isn’t “NHST with nicer words” — it needs a prior (a statement of what’s plausible before the data), and the Bayes factor can depend on that choice, so priors should be reasonable and reported. And a Bayes factor is relative evidence between the two models you specified — it says nothing about a third hypothesis you didn’t test. It’s a better tool for many questions, not a magic one.

Where this shows up next

Next, how to read the number Bayes reports — the Bayes factor — and then Bayesian versions of ANOVA and regression. See Chapter (Bayesian Methods).