Sampling: Random vs Convenience

Info sheet · Statistics for Psychology & Neuroscience

Info sheet 0.2 (draft) · Prerequisites: none · Give feedback ↗

Warning✎ Editing notes — to do / to check

Working notes for the author — not shown to students once collapsed; remove before publishing.

• Built from Chapter 5 (Sampling): full text, an interactive random-vs-convenience demo, code tabs, and a self-check. Version bumped to 0.2.
• (optional) add a small stratified-sampling worked example if space allows.

What you’ll get from this sheet

By the end you should be able to:

1. Distinguish random, stratified, and convenience sampling.
2. Say how the sampling method limits what you can generalise to.

Who you measure decides what you can conclude: a random sample supports generalisation to the population; a convenience sample (whoever’s easy to reach) risks bias you can’t undo with statistics.

From sample to population

We almost never measure a whole population — every adult with insomnia, every possible trial of a neuron. Instead we measure a sample and hope what we find generalises. Whether it can depends entirely on how the sample was drawn, and that decision is made before a single number is collected. No test performed afterwards can rescue a badly chosen sample; statistics can only quantify uncertainty within the sample you actually have.

Three ways to sample

• Random sampling — every member of the population has an equal chance of selection. This is the gold standard: it makes the sample representative on average and licenses honest generalisation to the whole population.
• Stratified sampling — split the population into subgroups (strata) — age bands, sites, conditions — and sample within each, guaranteeing every group appears in the proportion you want. Handy when a simple random draw might, by luck, under-represent a small but important group.
• Convenience sampling — you measure whoever is easy to reach: the undergraduates who signed up, the patients at one clinic. It’s cheap and common, but easy-to-reach people are usually systematically different from those who aren’t. Much of psychology’s “WEIRD” problem — samples that are Western, Educated, Industrialised, Rich, and Democratic — is convenience sampling in disguise.

Bias vs sampling error — and why n won’t save you

Two things pull a sample estimate away from the truth, and they behave completely differently. Sampling error is random: a particular sample lands a little above or below the true value by chance, and this shrinks as the sample grows — bigger n, tighter estimate. Bias is systematic: the sampling method points the estimate in one direction, and this does not shrink with n. A larger biased sample just gives you a more precise wrong answer. Pick a method and grow n — watch which error disappears and which one won’t:

md`**Sampling method**`

viewof mode = html`<select style="font-size:15px;padding:4px 8px"><option value="random" selected>Random (unbiased)</option><option value="conv">Convenience (biased subgroup)</option></select>`
md`**Sample size n** — currently **${+nn}**`

viewof nn = html`<input type="range" min="5" max="500" step="5" value="60" style="width:340px">`

sampWidget = {
  let s = 11;
  const rnd = () => { s|=0; s=s+0x6D2B79F5|0; let t=Math.imul(s^s>>>15,1|s); t=t+Math.imul(t^t>>>7,61|t)^t; return ((t^t>>>14)>>>0)/4294967296; };
  const trueMean = 50, convMean = 35;                 // easy-to-reach subgroup sits lower
  const drawExp = m => -m*Math.log(1-rnd());
  const n=+nn, data=[];
  for (let i=0;i<n;i++) data.push(drawExp(mode==="random"?trueMean:convMean));
  const mean = data.reduce((a,b)=>a+b,0)/n, bias = mean - trueMean;
  const W=470,H=280,padL=28,padR=14,padT=14,padB=30, lo=0, hi=180;
  const X=x=>padL+(Math.min(x,hi)-lo)/(hi-lo)*(W-padL-padR);
  const bins=30, bw=(hi-lo)/bins, c=Array(bins).fill(0);
  data.forEach(v=>{let b=Math.floor((v-lo)/bw); if(b>=0&&b<bins)c[b]++;});
  const mx=Math.max(1,...c), base=H-padB, plotH=H-padT-padB;
  const col = mode==="random" ? "#1F4E79" : "#C0501E";
  const bars=c.map((k,i)=>`<rect x="${X(lo+i*bw)}" y="${base-k/mx*plotH}" width="${(W-padL-padR)/bins-1}" height="${k/mx*plotH}" fill="${col}" fill-opacity="0.28"/>`).join("");
  const tLine=`<line x1="${X(trueMean)}" y1="${padT}" x2="${X(trueMean)}" y2="${base}" stroke="#1E7A34" stroke-width="2"/><text x="${X(trueMean)+4}" y="${padT+10}" font-size="10" fill="#1E7A34">true mean 50</text>`;
  const mLine=`<line x1="${X(mean)}" y1="${padT}" x2="${X(mean)}" y2="${base}" stroke="${col}" stroke-width="2" stroke-dasharray="4 3"/><text x="${X(mean)+4}" y="${padT+24}" font-size="10" fill="${col}">sample mean ${mean.toFixed(1)}</text>`;
  const axis=`<line x1="${padL}" y1="${base}" x2="${W-padR}" y2="${base}" stroke="#94a3b8"/>`;
  const verdict = mode==="random"
    ? `<span style="color:#1E7A34">Random: the estimate homes in on 50 as n grows — the gap is sampling <b>error</b>, and it shrinks.</span>`
    : `<span style="color:#c0392b">Convenience: the estimate is stuck ~${(-bias).toFixed(0)} below the truth. This <b>bias</b> does not shrink with n.</span>`;
  const fig=document.createElement("figure"); fig.style.margin="0";
  fig.innerHTML=`<svg viewBox="0 0 ${W} ${H}" width="${Math.min(W,540)}" style="background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px">${bars}${axis}${tLine}${mLine}</svg>`+
    `<figcaption style="margin-top:.5rem">n = <b>${n}</b> · sample mean = <b>${mean.toFixed(1)}</b> · bias = <b>${bias>=0?'+':''}${bias.toFixed(1)}</b>. ${verdict}</figcaption>`;
  return fig;
}

Under random sampling the estimate homes in on the true mean as n grows — that wobble is sampling error, and it melts away. Under convenience sampling the estimate settles a fixed distance from the truth and stays there however large the sample gets — that’s bias, and no amount of data removes it.

See it in code

R

Python

MATLAB / Octave

pop = exprnd(50, 1e5, 1);              % true mean ~50
mean(pop(randi(numel(pop), 60, 1)))    % random sample
easy = pop(pop < 45);                  % easy-to-reach subgroup
mean(easy(randi(numel(easy), 60, 1)))  % convenience sample: biased

The R and Python tabs run live; MATLAB is a static reference.

TipCheck your understanding

A researcher surveys students leaving the university gym about how much they exercise, and concludes the student body is very active. What kind of sample is this, and what can it actually generalise to?

It’s a convenience sample, and a badly biased one: people leaving a gym are, by definition, exercisers. The results generalise (at best) to gym-goers, not to the whole student body — and collecting more responses at the gym door wouldn’t fix it, it would just pin down the gym-goers’ average more precisely. To say anything about all students you’d need a random (or stratified) sample of the whole student population.

A big sample feels authoritative, but size only defeats sampling error, not bias. A biased method plus a huge n gives you a very precise wrong answer — and the resulting narrow confidence interval can make it look more trustworthy, not less. Always ask “who could not have ended up in this sample?” before “how many did I collect?”.

Where this shows up next

This is the first of the design decisions that determine what your statistics can claim. Next comes another: whether you manipulated a variable or merely observed it — the difference between association and cause.

Related topics

• Experiments vs Observation
• The Normal Distribution & the CLT
• It All Comes Down to Noise