t-Tests (One-Sample, Independent, Paired)

Info sheet · Statistics for Psychology & Neuroscience

Info sheet 0.2 (draft) · Prerequisites: the mean and SD; what a p-value is · Give feedback ↗

Warning✎ Editing notes — to do / to check

Working notes for the author — not shown to students once collapsed; remove before publishing.

• Added a “Did you know?” box on William Sealy Gosset / “Student” and the Guinness brewery.
• Added a 5-scenario radio quiz for choosing one-sample / independent / paired.
• Added an assumptions section (independence, normality, equal variances → Welch) and a robustness note.
• Added a “decide the tail before you look” section with its own 4-question one-vs-two-tailed quiz.

What you’ll get from this sheet

The workhorse for comparing two means — and why it doesn’t scale to many groups. By the end you should be able to:

1. Pick the right t‑test (one‑sample, independent, paired) and read the t‑formula.
2. Choose one‑ vs two‑tailed, and count pairwise comparisons with \(k(k-1)/2\).

A t‑test compares means by scaling the difference by its variability: \(t = \dfrac{\text{difference}}{\text{standard error}}\). Big difference or small variance → big t. Three flavours: one‑sample (mean vs a known value), independent (two unrelated groups), paired (two related scores). Comparing many groups needs \(k(k-1)/2\) t‑tests — which inflates family‑wise error.

NoteDid you know? The t-test was born in a brewery

The t-test was devised in 1908 by William Sealy Gosset, a chemist at the Guinness brewery in Dublin, who needed to draw conclusions from the small samples of an industrial process — barley varieties, yeast counts. Guinness treated statistical know-how as a trade secret and forbade employees from publishing, so Gosset wrote under the pseudonym “Student”, which is why we still call it Student’s t-distribution. Its heavier tails are precisely his correction for the extra uncertainty of small samples.

The three t-tests

All three ask “do these means differ?”, differing only in the design:

• One‑sample — compare a sample mean to a known value (is mean IQ different from 100?).
• Independent‑samples — compare two unrelated groups (danceability of 2000s vs 1990s songs).
• Paired‑samples — compare two related scores (the same people, pre vs post).

The engine is the same in each: the t‑statistic scales the difference in means by the variability. By hand, a large difference in means — or small variances — produces a large t, which infers a real difference; a small difference or large variances produces a small t, inferring none. It’s the signal‑to‑noise idea again: \(t = \text{signal} \div \text{noise}\).

Big difference or small noise → big t

Two decades of danceability. Slide the difference in their means and watch t and its p‑value respond; switch between a one‑ and two‑tailed test:

md`**Difference in means** — currently **${(+diff).toFixed(1)}**`

viewof diff = html`<input type="range" min="0" max="14" step="0.5" value="5" style="width:300px">`
md`**Test**`

viewof tail = html`<select style="font-size:15px;padding:4px 8px"><option value="two" selected>two-tailed</option><option value="one">one-tailed</option></select>`

tWidget = {
  const erf=x=>{const t=1/(1+0.3275911*Math.abs(x));const y=1-(((((1.061405429*t-1.453152027)*t)+1.421413741)*t-0.284496736)*t+0.254829592)*t*Math.exp(-x*x);return x<0?-y:y;};
  const Phi=z=>0.5*(1+erf(z/Math.SQRT2));
  const n=25, sd=10, m1=55, m2=55+ (+diff);
  const SE=sd*Math.sqrt(2/n), t=(m2-m1)/SE, df=2*n-2;
  const p = tail==="two" ? 2*(1-Phi(Math.abs(t))) : (1-Phi(t));
  const crit = tail==="two" ? 1.96 : 1.645;
  const W=460,H=220,pad=40, xmin=45,xmax=80;
  const X=x=>pad+(x-xmin)/(xmax-xmin)*(W-2*pad);
  const y1=90,y2=150;
  const grp=(y,m,col,lab)=>`<line x1="${X(m-SE)}" y1="${y}" x2="${X(m+SE)}" y2="${y}" stroke="${col}" stroke-width="2"/>`+
    `<circle cx="${X(m)}" cy="${y}" r="6" fill="${col}"/>`+
    `<text x="${X(m)}" y="${y-12}" text-anchor="middle" font-size="11" fill="${col}">${lab}: ${m.toFixed(1)}</text>`;
  const axis=`<line x1="${pad}" y1="${H-24}" x2="${W-pad}" y2="${H-24}" stroke="#94a3b8"/><text x="${W/2}" y="${H-8}" text-anchor="middle" font-size="11" fill="#667">danceability (mean ± SE)</text>`;
  const rej=p<0.05;
  const fig=document.createElement("figure"); fig.style.margin="0";
  fig.innerHTML=`<svg viewBox="0 0 ${W} ${H}" width="${Math.min(W,520)}" style="background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px">${grp(y1,m1,"#1F4E79","1990s")}${grp(y2,m2,"#C0501E","2000s")}${axis}</svg>`+
    `<figcaption style="margin-top:.5rem"><b>t(${df}) = <span style="color:${rej?'#1E7A34':'#c0392b'}">${t.toFixed(2)}</span></b>, ${tail}‑tailed p = <b style="color:${rej?'#1E7A34':'#c0392b'}">${p<0.0001?'< .0001':p.toFixed(4)}</b> (critical t ≈ ${crit}) → <b>${rej?'significant':'not significant'}</b>. A one‑tailed test puts all of α in one tail, so it's easier to reach — but only tests one direction.</figcaption>`;
  return fig;
}

At zero difference the means sit atop each other and t ≈ 0. Widen the gap (or, equivalently, shrink the noise) and t grows until it clears the critical value and the result turns significant. Flip to one‑tailed and the same t reaches significance sooner — because a one‑tailed test spends all of α (.05) in a single tail, testing only one direction (“2000s more danceable”), whereas a two‑tailed test splits α across both tails and is more conservative (testing for any difference).

One- or two-tailed? Decide before you look

A two-tailed test asks “is there any difference?” and splits α across both tails (±1.96 at α = .05). A one-tailed test asks a directional question (“is B greater than A?”), putting all of α in one tail (1.645), so it reaches significance more easily — but it is blind to an effect in the opposite direction. The non-negotiable rule: choose the direction before you see the data. Switching to one-tailed after peeking, because it drags you under .05, is a form of p-hacking. Reserve one-tailed tests for cases where a result in the other direction would be genuinely uninteresting or impossible.

NoteQuiz

Answer each question — the feedback appears beneath it.

tailBtn = {
  const scens=["Comparing two teaching methods with no prior direction in mind", "Testing whether a new painkiller differs from placebo (it could plausibly be worse)", "A pre-registered prediction that a fertiliser increases yield, where a decrease is commercially irrelevant", "You ran a two-tailed test, got p = .07, and want to switch to one-tailed to get under .05"];
  const ans=["Two-tailed", "Two-tailed", "One-tailed", "Two-tailed"];
  const why=["No directional prediction → test for any difference with a two-tailed test.", "The drug could help or harm, and you must be able to detect a harmful effect — two-tailed.", "A pre-registered, one-directional prediction where the opposite result is irrelevant justifies one-tailed.", "You can't switch tails after peeking — that's p-hacking. The honest test stays two-tailed."];
  const opts=["One-tailed", "Two-tailed"];
  const wrap=document.createElement("div");
  scens.forEach((scen,i)=>{
    const block=document.createElement("div"); block.style.cssText="margin:0 0 1rem 0";
    const q=document.createElement("div"); q.style.cssText="font-weight:600;margin-bottom:.35rem"; q.textContent=(i+1)+". "+scen;
    const row=document.createElement("div"); row.style.cssText="display:flex;gap:8px;flex-wrap:wrap;margin-bottom:.3rem";
    const fb=document.createElement("div"); fb.style.cssText="font-size:14px;min-height:1.2rem;color:#94a3b8"; fb.textContent="Click an answer to check.";
    opts.forEach(opt=>{
      const b=document.createElement("button"); b.textContent=opt;
      b.style.cssText="padding:5px 11px;border:1px solid #cbd5e1;border-radius:6px;background:#fff;cursor:pointer;font-size:14px";
      b.onmouseenter=()=>b.style.background="#f1f5f9"; b.onmouseleave=()=>b.style.background="#fff";
      b.onclick=()=>{ const ok=opt===ans[i];
        fb.innerHTML=`<b style="color:${ok?'#1E7A34':'#c0392b'}">${ok?'✓ '+ans[i]:'✗ Not quite — '+ans[i]}.</b> <span style="color:#334">${why[i]}</span>`; };
      row.appendChild(b);
    });
    block.append(q,row,fb); wrap.appendChild(block);
  });
  return wrap;
}

Assumptions — and how robust the t-test is

The t-test rests on three assumptions:

• Independence of observations — each measurement is separate and uninfluenced by the others. This is the critical one: violate it (e.g. analysing repeated measures as if they were independent) and the test is simply wrong. The paired test is the designed-in way to handle dependence, by analysing the differences.
• Normality — of the underlying data, or more precisely of the sampling distribution of the mean.
• Equal variances — for the independent t-test, the two groups are assumed to have similar spread. When they don’t, use Welch’s t-test (the default in R’s t.test()), which adjusts the degrees of freedom.

The reassuring part is robustness: thanks to the CLT, the t-test tolerates non-normality well once samples are reasonably large (roughly n ≳ 30 per group), because it’s the mean’s distribution that needs to be normal, not the raw data. The danger zone is small samples of clearly non-normal data — there, consider a transformation or a non-parametric alternative (Mann–Whitney / Wilcoxon).

Which t-test? Quick quiz

Match each scenario to the right t-test:

NoteQuiz

Answer each question — the feedback appears beneath it.

whichBtn = {
  const scens=["Is your sample's mean IQ different from the population value of 100?", "Do 2000s songs differ in danceability from 1990s songs (different songs)?", "Do the same patients' anxiety scores change from before to after therapy?", "Is a class's average exam score different from the national average of 65?", "Do left-handers and right-handers differ in reaction time (separate people)?"];
  const ans=["One-sample", "Independent", "Paired", "One-sample", "Independent"];
  const why=["Comparing one sample's mean to a known fixed value (100) → one-sample t-test.", "Two unrelated groups of different songs → independent-samples t-test.", "The same people measured twice (pre vs post) → paired-samples t-test.", "One group's mean against a fixed benchmark (65) → one-sample t-test.", "Two separate groups of people (left- vs right-handers) → independent-samples t-test."];
  const opts=["One-sample", "Independent", "Paired"];
  const wrap=document.createElement("div");
  scens.forEach((scen,i)=>{
    const block=document.createElement("div"); block.style.cssText="margin:0 0 1rem 0";
    const q=document.createElement("div"); q.style.cssText="font-weight:600;margin-bottom:.35rem"; q.textContent=(i+1)+". "+scen;
    const row=document.createElement("div"); row.style.cssText="display:flex;gap:8px;flex-wrap:wrap;margin-bottom:.3rem";
    const fb=document.createElement("div"); fb.style.cssText="font-size:14px;min-height:1.2rem;color:#94a3b8"; fb.textContent="Click an answer to check.";
    opts.forEach(opt=>{
      const b=document.createElement("button"); b.textContent=opt;
      b.style.cssText="padding:5px 11px;border:1px solid #cbd5e1;border-radius:6px;background:#fff;cursor:pointer;font-size:14px";
      b.onmouseenter=()=>b.style.background="#f1f5f9"; b.onmouseleave=()=>b.style.background="#fff";
      b.onclick=()=>{ const ok=opt===ans[i];
        fb.innerHTML=`<b style="color:${ok?'#1E7A34':'#c0392b'}">${ok?'✓ '+ans[i]:'✗ Not quite — '+ans[i]}.</b> <span style="color:#334">${why[i]}</span>`; };
      row.appendChild(b);
    });
    block.append(q,row,fb); wrap.appendChild(block);
  });
  return wrap;
}

Why it doesn’t scale: pairwise comparisons

A t‑test compares two groups. To compare more, you’d need every pair — and the count grows fast:

\[\text{number of comparisons} = \frac{k(k-1)}{2}\]

So 3 groups → 3 tests, 4 → 6, and 6 groups → 15. Each test carries its own 1‑in‑20 chance of a false positive at α = .05, and those chances accumulate — much like rolling a die ten times gives roughly an 84% chance of at least one six. That runaway family‑wise error is why we don’t just spray t‑tests at a multi‑group design; we use ANOVA (one omnibus test) and, if needed, corrected post‑hoc comparisons.

See it in code

R

Python

MATLAB / Octave

[h, p, ci, stats] = ttest2(d00, d90);   % independent-samples t-test
% [h,p] = ttest(x, 50);                 % one-sample
% [h,p] = ttest(pre, post);             % paired

The R and Python tabs run live; MATLAB is a static reference.

TipCheck your understanding

You compare two groups and get t = 0.4, p = .69, even though their means differ by a few points. Why isn’t it significant — and would a one‑tailed test have saved it?

Because t is the difference relative to its standard error: a few points of difference against large within‑group variability (or a small sample) gives a small t. The signal is swamped by noise. A one‑tailed test would only help modestly (it lowers the threshold from ~1.96 to ~1.64), and only if the difference is in the predicted direction — it can’t rescue a t of 0.4. The real fix is more data or less noise, not a friendlier tail.

Decide one‑ vs two‑tailed before you see the data — picking a one‑tailed test after peeking, because it nudges you under .05, is a form of p‑hacking. A one‑tailed test also can’t detect an effect in the opposite direction, so use it only when a reversal would be uninteresting or impossible. And never answer a many‑group question with a pile of t‑tests — the \(k(k-1)/2\) comparisons inflate family‑wise error; use ANOVA instead.

Where this shows up next

The pairwise‑comparison explosion motivates Family‑Wise Error and ANOVA (one test for many groups). And the independent t‑test is just a two‑group general linear model. See Chapter (Inference).

Related topics

• What a p-value Actually Is — the number a t‑test produces.
• Family-Wise Error — why many pairwise tests go wrong.
• One-Way ANOVA & the F-ratio — one test for many groups.
• The General Linear Model — a t‑test is a GLM with one binary predictor.
• It All Comes Down to Noise — t = signal ÷ noise.