Experiments vs Observation (Why Correlation ≠ Causation)

Info sheet · Statistics for Psychology & Neuroscience

Info sheet 0.2 (draft) · Prerequisites: sampling · Give feedback ↗

Warning✎ Editing notes — to do / to check

Working notes for the author — not shown to students once collapsed; remove before publishing.

• Built from Chapter 5 (Experiments & Observations): full text, a confounder/randomisation demo, code tabs, and a self-check. Version bumped to 0.2.

What you’ll get from this sheet

By the end you should be able to:

1. Tell an experiment from an observational study.
2. Explain why only randomised manipulation licenses causal claims.

Only a manipulated, randomised variable supports a causal claim; in observational data, a lurking third variable can always explain an association — correlation is not causation.

Two kinds of study, two kinds of claim

Studies come in two broad kinds, and the distinction decides what you’re allowed to conclude. In an experiment, the researcher manipulates an independent variable and randomly assigns participants to conditions. Those two ingredients — manipulation and randomisation — are what let us infer causation: random assignment makes the groups equivalent, on average, on everything except the thing we changed, so any difference in the outcome can be pinned on the manipulation.

In an observational study we simply measure variables as they naturally occur — we can’t randomly assign people to be depressed, or to have a particular brain structure, or to drink six coffees a day. Because we didn’t intervene, we’re generally limited to claims about association. This is the deep reason behind the slogan correlation is not causation: whenever two things move together in observational data, a lurking third variable (a confounder) might be driving both.

Why randomisation is the magic ingredient

A confounder is a variable that influences both the thing you measured and the outcome, manufacturing an association that isn’t causal. Stress, say, might raise both coffee intake and anxiety, making coffee and anxiety correlate even if coffee does nothing. Randomisation breaks that link: if you assign coffee doses at random, intake no longer depends on stress (or anything else), so a leftover association must be causal. Flip between an observational study and a randomised experiment and watch what happens to an apparent effect when the true causal effect is actually zero:

md`**Study type**`

viewof design = html`<select style="font-size:15px;padding:4px 8px"><option value="obs" selected>Observational (X just measured)</option><option value="exp">Experiment (X randomised)</option></select>`
md`**Confounder strength** — currently **${(+conf).toFixed(1)}**`

viewof conf = html`<input type="range" min="0" max="3" step="0.1" value="1.6" style="width:340px">`

confWidget = {
  let s = 3;
  const rnd=()=>{s|=0;s=s+0x6D2B79F5|0;let t=Math.imul(s^s>>>15,1|s);t=t+Math.imul(t^t>>>7,61|t)^t;return((t^t>>>14)>>>0)/4294967296;};
  const nrm=()=>{const u=1-rnd(),v=rnd();return Math.sqrt(-2*Math.log(u))*Math.cos(2*Math.PI*v);};
  const n=80, c=+conf, xs=[], ys=[];
  for(let i=0;i<n;i++){
    const z=nrm();                                       // hidden confounder
    const x = design==="exp" ? nrm() : z + 0.5*nrm();    // randomised: X ⟂ Z; observational: X tracks Z
    const y = 0*x + c*z + nrm();                         // TRUE causal effect of X on Y is 0
    xs.push(x); ys.push(y);
  }
  const mx=xs.reduce((a,b)=>a+b,0)/n, my=ys.reduce((a,b)=>a+b,0)/n;
  let Sxy=0,Sxx=0,Syy=0; for(let i=0;i<n;i++){Sxy+=(xs[i]-mx)*(ys[i]-my);Sxx+=(xs[i]-mx)**2;Syy+=(ys[i]-my)**2;}
  const slope=Sxy/Sxx, r=Sxy/Math.sqrt(Sxx*Syy), b0=my-slope*mx;
  const W=460,H=280,pad=34, xmin=-3.2,xmax=3.2,ymin=-5,ymax=5;
  const X=x=>pad+(x-xmin)/(xmax-xmin)*(W-2*pad), Y=y=>H-pad-(y-ymin)/(ymax-ymin)*(H-2*pad);
  const dots=xs.map((x,i)=>`<circle cx="${X(x)}" cy="${Y(ys[i])}" r="3" fill="#1f2937" fill-opacity="0.55"/>`).join("");
  const line=`<line x1="${X(xmin)}" y1="${Y(b0+slope*xmin)}" x2="${X(xmax)}" y2="${Y(b0+slope*xmax)}" stroke="#C0501E" stroke-width="2.5"/>`;
  const axis=`<line x1="${pad}" y1="${Y(0)}" x2="${W-pad}" y2="${Y(0)}" stroke="#cbd5e1"/><line x1="${X(0)}" y1="${pad}" x2="${X(0)}" y2="${H-pad}" stroke="#cbd5e1"/>`;
  const verdict = design==="exp"
    ? `<span style="color:#1E7A34">Randomised: X is independent of the confounder, so the estimated slope ≈ 0 — matching the <b>true</b> zero causal effect.</span>`
    : `<span style="color:#c0392b">Observational: X and Y look related (slope ${slope.toFixed(2)}, r ${r.toFixed(2)}), yet X has <b>no</b> causal effect — the confounder drives both. Correlation ≠ causation.</span>`;
  const fig=document.createElement("figure"); fig.style.margin="0";
  fig.innerHTML=`<svg viewBox="0 0 ${W} ${H}" width="${Math.min(W,540)}" style="background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px">${axis}${dots}${line}</svg>`+
    `<figcaption style="margin-top:.5rem">true causal effect of X = <b>0</b> · estimated slope = <b style="color:#C0501E">${slope.toFixed(2)}</b> · r = <b>${r.toFixed(2)}</b>. ${verdict}</figcaption>`;
  return fig;
}

In the observational view, turning up the confounder manufactures a convincing slope out of nothing — X and Y march together even though X does nothing. Switch to the experiment and the slope collapses to zero however strong the confounder is, because randomising X severs its link to the confounder. That collapse is the value of an experiment.

Where you can’t randomise — you can’t assign people to smoke, or to a childhood — natural and quasi-experiments exploit accidents of circumstance (a policy change, a lottery) to approximate random assignment, but they always carry more assumptions than a true experiment.

See it in code

R

Python

MATLAB / Octave

rng(1); n = 500;
z = randn(n,1);                       % hidden confounder; true effect of X is 0
x_obs = z + 0.5*randn(n,1);           % observational: X tracks the confounder
y  = 0*x_obs + 1.6*z + randn(n,1);
b = polyfit(x_obs, y, 1);  b(1)       % biased slope

x_exp = randn(n,1);                    % randomised
y2 = 0*x_exp + 1.6*z + randn(n,1);
b2 = polyfit(x_exp, y2, 1);  b2(1)     % ~0 — the truth

The R and Python tabs run live; MATLAB is a static reference.

TipCheck your understanding

A study finds that people who drink more coffee report more anxiety, and a headline concludes “coffee causes anxiety.” What’s the flaw, and what would settle it?

The study is observational, so the association could be produced by a confounder — stress or workload might raise both coffee intake and anxiety — or the causation could even run backwards (anxious people drink more coffee to cope). Observational data can’t separate these accounts. Only a randomised experiment, assigning people to different coffee doses so that intake is independent of stress and everything else, licenses the causal claim.

“Controlling for” measured covariates in observational data is not the same as randomising. You can only adjust for confounders you thought of and measured; randomisation balances the ones you never even considered. So an observational study with a long list of covariates is still not an experiment — unmeasured confounding is always the standing threat.

Where this shows up next

That lurking third variable returns as the engine of mediation. And once you’ve settled what you’re comparing, the next design choice is who — the same people across conditions, or different ones.

Related topics

• Mediation (the lurking third variable)
• Sampling: Random vs Convenience
• Within- vs Between-Subjects Designs