Evaluating Models (R², MSE, AIC)
Info sheet · Statistics for Psychology & Neuroscience
Warning✎ Editing notes — to do / to check
Working notes for the author — not shown to students once collapsed; remove before publishing.
- [ ]
What you’ll get from this sheet
How to tell a good model from a bad one — beyond “is it significant?”. By the end you should be able to:
- Name four families of model metrics and what each measures.
- Explain why R² always rises with more predictors, and why adjusted R² and AIC don’t.
Judge a model on several axes: F/p (is it better than chance?), MSE (how close are the predictions?), R² (share of variance explained — but it only ever rises as you add predictors), adjusted R² and AIC (which penalise complexity, so they reward parsimony).
Four families of metrics
Regression is for prediction, not just detecting differences — so “garbage in, garbage out” bites hard, and we need ways to tell good from bad:
- F‑statistic & p‑value — reliability. Is the model better than chance? F = variance explained ÷ variance unexplained; a large F with p < .05 says yes.
- Residuals & Mean Squared Error (MSE) — prediction error. \(\text{MSE} = \frac{1}{n}\sum (y_i - \hat y_i)^2\); lower is better, and it’s an easy way to compare models.
- R² & adjusted R² — variance explained. R² is the proportion of variance the model accounts for (0.5 = half explained, quite good; 1.0 = perfect, and usually suspicious). Its flaw: R² always goes up when you add a predictor, even a meaningless one. Adjusted R² corrects for that by penalising extra predictors.
- AIC (Akaike’s Information Criterion) — balances fit against the number of predictors; lower AIC = better trade‑off.
Watch the metrics disagree
Here’s a dataset where only the first two predictors are real; the rest are pure noise. Add predictors one at a time and watch R² keep creeping up — while adjusted R² peaks and then falls, and AIC bottoms out, both pointing at the true two‑predictor model.
Compare the numbers as you slide. R² inches upward with every predictor you add — it can’t tell signal from noise, because more parameters always let the line hug the sample a little closer. Adjusted R² and AIC are wiser: they charge a fee for each predictor, so they improve while you’re adding real signal and then reverse once you start adding junk. Both flag the same two‑predictor model as best — which is exactly the parsimony the last sheet argued for.
See it in code
m2 = fitlm(d, 'y ~ x1 + x2');
m4 = fitlm(d, 'y ~ x1 + x2 + x3 + x4');
[m2.Rsquared.Adjusted, m4.Rsquared.Adjusted] % adjusted R^2
[m2.ModelCriterion.AIC, m4.ModelCriterion.AIC] % AIC (lower is better)The R and Python tabs run live; MATLAB is a static reference.
TipCheck your understanding
Model B has a higher R² than model A but also a higher AIC. Which is better?
Probably A. R² always rises (or stays flat) when you add predictors, so B’s higher R² may just be extra parameters hugging the sample. AIC penalises that complexity, and lower AIC is better — so a higher AIC for B says its improved fit doesn’t justify its extra predictors. Trust the complexity‑aware measures (adjusted R², AIC) over raw R² when comparing models of different sizes.
No single number settles it. A high R² can reflect overfitting, not quality; a low p only says “better than chance”, not “good”; MSE depends on the scale of y (so it’s for comparing models on the same data, not an absolute standard); and AIC is only comparable between models fit to the same dataset. Report several, and pair them with the diagnostic plots and out‑of‑sample checks — reliability and fit are different questions.
Where this shows up next
This closes the regression cluster. The same evaluate‑and‑compare thinking returns for mixed models and the Bayesian alternative (Bayes factors). See Chapter (Model Building & Evaluation).