Choosing Variables (Over- vs Under-fitting)

Info sheet · Statistics for Psychology & Neuroscience

Author

Andrew Bell

Published

August 13, 2026

Info sheet 0.2 (draft) · Prerequisites: multiple regression; residuals and MSE · Give feedback ↗

✎ Editing notes — to do / to check

Working notes for the author — not shown to students once collapsed; remove before publishing.

What you’ll get from this sheet

How many predictors should a model have? By the end you should be able to:

Explain the trade‑off between too many and too few predictors.
State the law of parsimony (Occam’s razor) and why it guides variable selection.

Too few predictors → underfitting (high bias, misses real structure). Too many → overfitting (high variance, chases noise, won’t generalise). The sweet spot is parsimony (Occam’s razor): the fewest predictors that do the job.

Too many vs too few

Say we’re modelling median house value from 13 possible predictors. Should we use all 13? It’s not obvious, because there are costs on both sides. Adding predictors always improves the fit to this sample — but at a price: overfitting, collinearity, and a hunger for more data (statistical power). Stripping predictors too far risks underfitting — a model so simple it explains almost nothing and misses real structure.

This is the bias–variance trade‑off: too‑simple models are biased (systematically wrong), too‑complex models have high variance (they swing around to fit noise, so they change wildly from sample to sample). Somewhere between sits the sweet spot.

Watch a model over‑fit

Below, the black points are the training data. Slide the model’s flexibility (polynomial degree) up and it fits those points ever more tightly — but watch the test error (on new data it hasn’t seen). Training error only ever falls; test error falls, bottoms out, then climbs as the model starts memorising noise.

md`**Model flexibility (polynomial degree)** — currently **${+deg}**`

viewof deg = html`<input type="range" min="1" max="8" step="1" value="1" style="width:340px">`

overfitWidget = {
  const solve=(A,b)=>{const n=b.length,M=A.map((r,i)=>r.concat(b[i]));
    for(let c=0;c<n;c++){let p=c;for(let r=c+1;r<n;r++)if(Math.abs(M[r][c])>Math.abs(M[p][c]))p=r;[M[c],M[p]]=[M[p],M[c]];
      const pv=M[c][c]||1e-9;for(let r=0;r<n;r++)if(r!==c){const f=M[r][c]/pv;for(let k=c;k<=n;k++)M[r][k]-=f*M[c][k];}}
    return M.map((r,i)=>r[n]/(M[i][i]||1e-9));};
  const polyfit=(xs,ys,d)=>{const X=xs.map(x=>Array.from({length:d+1},(_,k)=>x**k));
    const A=Array.from({length:d+1},(_,i)=>Array.from({length:d+1},(_,j)=>X.reduce((s,r)=>s+r[i]*r[j],0)));
    const b=Array.from({length:d+1},(_,i)=>X.reduce((s,r,k)=>s+r[i]*ys[k],0)); return solve(A,b);};
  const pred=(c,x)=>c.reduce((s,ci,k)=>s+ci*x**k,0);
  const truef=x=>0.8*x*x+0.5*x+0.2;
  let s=21; const rnd=()=>{s|=0;s=s+0x6D2B79F5|0;let t=Math.imul(s^s>>>15,1|s);t=t+Math.imul(t^t>>>7,61|t)^t;return((t^t>>>14)>>>0)/4294967296;};
  const nrm=()=>{const u=1-rnd(),v=rnd();return Math.sqrt(-2*Math.log(u))*Math.cos(2*Math.PI*v);};
  const ntr=12, nte=40, trX=[],trY=[],teX=[],teY=[];
  for(let i=0;i<ntr;i++){const x=-1+2*i/(ntr-1); trX.push(x); trY.push(truef(x)+0.28*nrm());}
  for(let i=0;i<nte;i++){const x=-1+2*i/(nte-1); teX.push(x); teY.push(truef(x)+0.28*nrm());}

  const d=+deg, c=polyfit(trX,trY,d);
  const mse=(xs,ys)=>xs.reduce((sm,x,i)=>sm+(ys[i]-pred(c,x))**2,0)/xs.length;
  const trMSE=mse(trX,trY), teMSE=mse(teX,teY);

  const W=460,H=300,pad=32, xmin=-1.05,xmax=1.05,ymin=-0.6,ymax=2.2;
  const X=x=>pad+(x-xmin)/(xmax-xmin)*(W-2*pad), Y=y=>H-pad-(y-ymin)/(ymax-ymin)*(H-2*pad);
  let curve="M "; for(let i=0;i<=120;i++){const x=xmin+(xmax-xmin)*i/120; curve+=`${X(x)} ${Y(Math.max(ymin,Math.min(ymax,pred(c,x))))} ${i<120?'L ':''}`;}
  let truep="M "; for(let i=0;i<=120;i++){const x=xmin+(xmax-xmin)*i/120; truep+=`${X(x)} ${Y(truef(x))} ${i<120?'L ':''}`;}
  let dots=""; trX.forEach((x,i)=>dots+=`<circle cx="${X(x)}" cy="${Y(trY[i])}" r="3.5" fill="#1f2937"/>`);
  const axes=`<line x1="${pad}" y1="${H-pad}" x2="${W-pad}" y2="${H-pad}" stroke="#94a3b8"/><line x1="${pad}" y1="${pad}" x2="${pad}" y2="${H-pad}" stroke="#94a3b8"/>`;
  const verdict = d<2 ? ["underfitting — too rigid to capture the curve (bias)","#C0501E"]
                : d<=3 ? ["a good fit — captures the trend without chasing noise","#1E7A34"]
                : ["overfitting — wiggling to hit training points, worse on new data (variance)","#c0392b"];
  const fig=document.createElement("figure"); fig.style.margin="0";
  fig.innerHTML=`<svg viewBox="0 0 ${W} ${H}" width="${Math.min(W,520)}" style="background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px">${axes}`+
    `<path d="${truep}" fill="none" stroke="#94a3b8" stroke-dasharray="4 3"/>`+
    `<path d="${curve}" fill="none" stroke="#1F4E79" stroke-width="2.5"/>${dots}</svg>`+
    `<figcaption style="margin-top:.5rem">training MSE = <b style="color:#1F4E79">${trMSE.toFixed(3)}</b> · test MSE = <b style="color:${verdict[1]}">${teMSE.toFixed(3)}</b>. `+
    `<b style="color:${verdict[1]}">${verdict[0]}.</b> <span style="color:#94a3b8">(grey dashed = true relationship)</span></figcaption>`;
  return fig;
}

At degree 1 the line is too stiff to follow the gentle curve — it underfits, and both errors are high (that’s bias). Around degree 2–3 it traces the true relationship and the test error is lowest. Push it higher and the curve starts snaking through individual training points: the training error keeps shrinking, but the test error climbs, because the model is now fitting noise it will never see again. That’s overfitting — and it’s why more predictors isn’t automatically better.

The law of parsimony

The guide is Occam’s razor: the best model is as complex as it needs to be, and no more. In regression that means neither throwing in every predictor you can find, nor stripping it down until it explains nothing — but finding the minimum set of predictors that meets the goal. Variable selection isn’t arbitrary; it’s this balance between simplicity and explanatory power. The next sheet gives you the metrics (adjusted R², AIC) that put numbers on it.

See it in code

x = linspace(-1,1,12)';  y = 0.8*x.^2 + 0.5*x + 0.2 + 0.28*randn(12,1);
for d = [1 2 6]
  c = polyfit(x, y, d);
  % evaluate polyval(c, xnew) against held-out ynew ...
end

The R and Python tabs run live; MATLAB is a static reference.

Check your understanding

Adding a predictor makes your R² go up. Does that mean you should keep it?

Not necessarily. R² (and the training fit generally) always improves — or at least never worsens — when you add a predictor, even a useless one, because the model has more freedom to chase the sample. The real question is whether it improves fit on new data. Use a complexity‑aware measure (adjusted R², AIC) or held‑out/cross‑validated error, and Occam’s razor: keep the predictor only if it earns its place.

“Fit to the sample” and “quality of the model” are not the same thing — a model can fit the training data perfectly and still predict terribly. Beware automated stepwise selection run without thought (it capitalises on chance and inflates significance), and remember that adding predictors also demands more data and can introduce multicollinearity. Simpler models are also easier to interpret and replicate.

Where this shows up next

Next: the metrics that quantify this balance — MSE, R², adjusted R², and AIC. See Chapter (Model Building & Evaluation).

What you’ll get from this sheet

Too many vs too few

Watch a model over‑fit

The law of parsimony

See it in code

Where this shows up next

Related topics