Why Not Just Run Lots of One-Way ANOVAs?

Info sheet · Statistics for Psychology & Neuroscience

Info sheet 0.2 (draft) · Prerequisites: one-way and factorial ANOVA; the F-ratio · Give feedback ↗

Warning✎ Editing notes — to do / to check

Working notes for the author — not shown to students once collapsed; remove before publishing.

• [ ]

What you’ll get from this sheet

A straightforward question: why not just run multiple one-way ANOVAs instead of a single factorial ANOVA? At first glance both approaches seem to give the same information — but the difference reveals a fundamental principle of the general linear model. By the end you should be able to:

1. Say why a factorial ANOVA beats several one-way ANOVAs (or a pile of t-tests).
2. Explain the relationship between model variance and residual variance in a GLM.

Ignore a real factor and its variance gets dumped into the residual — inflating your error term, shrinking every F, and hiding effects. A factorial ANOVA gives each factor its own term, so the residual is just noise.

One-way vs factorial

Recall that in a two-way factorial ANOVA we’re essentially asking two questions: does IV1 affect the DV, and does IV2 affect the DV? You might think you could answer these with two separate one-way ANOVAs, each collapsing across the other factor. Run that on the mini-Netflix data (360 movies) and you do get a significant main effect of Genre, and a significant main effect of Rating. So far, so good.

But compare them to the two-way factorial ANOVA and the F-values and p-values differ — the factorial version comes back stronger. Why?

The key difference: residual variance

The F-ratio is the Mean Square for the model over the Mean Square for the residuals. So anything that bloats the residual term drives F down.

• In a one-way ANOVA on Genre, any variability due to the ignored factor (Rating) has nowhere to go but the residual — artificially inflating residual variance.
• In a factorial ANOVA, each factor gets its own model term. The residual is left holding only genuine, unexplained noise.

Slide the strength of the ignored factor (Rating) up and down. The Genre effect stays exactly the same — but watch what happens to its F when you analyse Genre alone versus inside a factorial model:

md`**How strongly the *ignored* factor (Rating) affects the outcome** — currently **${(+rEff).toFixed(1)}**`

viewof rEff = html`<input type="range" min="0" max="14" step="0.5" value="8" style="width:340px">`

partitionWidget = {
  const nCell=12, k=4, N=nCell*k;      // 2x2 design, 12 per cell
  const gEffect=4, noiseSD=6;          // fixed Genre effect + fixed noise
  // fixed-seed normals
  let s=9;
  const rand=()=>{s|=0;s=s+0x6D2B79F5|0;let t=Math.imul(s^s>>>15,1|s);t=t+Math.imul(t^t>>>7,61|t)^t;return((t^t>>>14)>>>0)/4294967296;};
  const z=[]; for(let i=0;i<N;i++){const u=1-rand(),v=rand();z.push(Math.sqrt(-2*Math.log(u))*Math.cos(2*Math.PI*v));}

  const R=+rEff;
  const gc=[-1,-1,1,1], rc=[-1,1,-1,1];   // cell codes: genre, rating
  const val=[], cellOf=[], genreOf=[];
  for(let c=0;c<k;c++) for(let j=0;j<nCell;j++){ const i=c*nCell+j; val.push(55 + gEffect*gc[c] + R*rc[c] + noiseSD*z[i]); cellOf.push(c); genreOf.push(gc[c]); }
  const grand=val.reduce((a,b)=>a+b,0)/N;
  // genre means (24 each)
  let gSum=[0,0], gN=[0,0];
  val.forEach((y,i)=>{ const gi=genreOf[i]<0?0:1; gSum[gi]+=y; gN[gi]++; });
  const gMean=[gSum[0]/gN[0], gSum[1]/gN[1]];
  const SS_genre = gN[0]*(gMean[0]-grand)**2 + gN[1]*(gMean[1]-grand)**2;
  const MS_genre = SS_genre/1;
  // cell means (factorial residual = within cell)
  const cSum=[0,0,0,0], cN=[0,0,0,0];
  val.forEach((y,i)=>{ cSum[cellOf[i]]+=y; cN[cellOf[i]]++; });
  const cMean=cSum.map((s,i)=>s/cN[i]);
  let SS_res_fact=0; val.forEach((y,i)=>SS_res_fact+=(y-cMean[cellOf[i]])**2);
  const MS_res_fact=SS_res_fact/(N-4);
  // one-way residual (within genre = includes rating variance)
  let SS_res_1way=0; val.forEach((y,i)=>{ const gi=genreOf[i]<0?0:1; SS_res_1way+=(y-gMean[gi])**2; });
  const MS_res_1way=SS_res_1way/(N-2);
  const F_fact=MS_genre/MS_res_fact, F_1way=MS_genre/MS_res_1way;
  const Fcrit=4.06;

  const W=440,H=200,pad=20;
  const bmax=H-2*pad-20, sc=bmax/Math.max(MS_res_1way,MS_res_fact,1);
  const bx0=60, bw=70, gap=150;
  const bar=(x,val2,col,lab)=>`<rect x="${x}" y="${H-pad-20-val2*sc}" width="${bw}" height="${val2*sc}" fill="${col}"/>`+
    `<text x="${x+bw/2}" y="${H-pad-4}" text-anchor="middle" font-size="11" fill="#334">${lab}</text>`+
    `<text x="${x+bw/2}" y="${H-pad-24-val2*sc}" text-anchor="middle" font-size="10" fill="#334">MS = ${val2.toFixed(0)}</text>`;
  const fig=document.createElement("figure"); fig.style.margin="0";
  fig.innerHTML=
    `<svg viewBox="0 0 ${W} ${H}" width="${Math.min(W,520)}" style="background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px">`+
      `<text x="${W/2}" y="16" text-anchor="middle" font-size="12" fill="#667">residual (error) mean square</text>`+
      bar(bx0,MS_res_1way,"#C0501E","one-way on Genre")+bar(bx0+gap,MS_res_fact,"#1E7A34","factorial")+`</svg>`+
    `<figcaption style="margin-top:.5rem">Same Genre effect (MS = ${MS_genre.toFixed(0)}), two error terms:<br>`+
    `one-way <b style="color:#C0501E">F(1,${N-2}) = ${F_1way.toFixed(2)}</b> vs factorial <b style="color:#1E7A34">F(1,${N-4}) = ${F_fact.toFixed(2)}</b> `+
    `(critical F ≈ ${Fcrit}). Ignoring Rating dumps its variance into the residual, dragging F down.</figcaption>`;
  return fig;
}

Set the ignored factor to zero and the two F-values almost coincide — with nothing to ignore, it doesn’t matter. But turn Rating’s effect up and the two diverge fast: the factorial F barely moves (its residual is still just noise), while the one-way F sinks, because Rating’s variance is now piled into its error term. Same signal, weaker test — purely because we ignored a real factor.

The GLM perspective

This ties straight back to the general linear model: Outcome = Model + Error. The Model term is the variance we explain with the factors we include; the Error term is what’s left. Every relevant factor you add to the model is variance removed from the error — so a factorial ANOVA, by modelling both factors, shrinks the error and delivers a stronger, more reliable test. Factorial → smaller residual → larger F → lower p. Multiple one-way ANOVAs → larger residual → smaller F → higher p.

The extra benefit: interactions

The greatest advantage of a factorial ANOVA isn’t the tighter error term, though — it’s that it can test interactions. Does the effect of Age Rating on Rotten Tomatoes depend on the Genre? Perhaps action films rated “7+” score higher than dramas rated “7+.” Effects like that are invisible to separate one-way ANOVAs — interactions are unique to factorial designs, and they’re often the most interesting findings in the whole analysis.

See it in code

The same Genre effect, tested two ways — note how the residual (and so the F) changes:

R

Python

MATLAB / Octave

genre  = [ones(1,24) 2*ones(1,24)];              % Drama / Comedy
rating = repmat([ones(1,12) 2*ones(1,12)], 1, 2);
score  = 55 + 4*(genre==2) + 8*(rating==2) + 6*randn(1,48);
anovan(score, {genre},          'varnames', {'Genre'});             % one-way
anovan(score, {genre, rating}, 'model','interaction', ...          % factorial
       'varnames', {'Genre','Rating'});

Static reference — the R and Python tabs run live in the page.

TipCheck your understanding

Two one-way ANOVAs both come back significant. Why bother with the factorial version?

Three reasons. It gives a smaller residual (Rating’s variance is modelled, not dumped into error), so the tests are more powerful and the F-values more accurate. It controls Type I error better than running several separate tests. And — most importantly — it can detect an interaction, which no set of one-way ANOVAs can.

This only works if the extra factor is real. Adding an irrelevant factor spends a degree of freedom without soaking up meaningful variance, which can nudge the residual MS the wrong way. Include factors because your design has them, not to fish for a bigger F.

Where this shows up next

See the Factorial ANOVA & Interactions and Spotting & Visualising Interactions sheets for how to run and read those interaction effects, and Chapter 8 (ANOVA) for the full treatment.

Related topics

• Factorial ANOVA & Interactions — the model this argues for.
• Spotting & Visualising Interactions — the payoff a factorial design unlocks.
• The General Linear Model — Outcome = Model + Error, the principle at work.
• Family-Wise Error — another hazard of running many tests.