PCA in Action
Info sheet · Statistics for Psychology & Neuroscience
Warning✎ Editing notes — to do / to check
Working notes for the author — not shown to students once collapsed; remove before publishing.
- [ ]
What you’ll get from this sheet
The whole PCA pipeline on a familiar problem: predicting a song’s popularity from its audio features. By the end you should be able to:
- Decide how many components to keep (the Kaiser criterion).
- Say what you gain by regressing on components instead of raw predictors — and what you give up.
Run PCA, keep the components with an eigenvalue greater than 1 (the Kaiser criterion), and regress on those. You trade a sliver of explained variance for a far more parsimonious model with zero multicollinearity.
The pipeline
Let’s put PCA to work: predict a song’s popularity from its audio features. We first run an ordinary “baseline” regression on all the predictors, then run PCA, keep a few components, and regress on those instead — and compare.
We run the PCA and extract, for each song, its coordinates along each principal component (in R, FactoMineR and factoextra with get_pca_ind() pull out those individual coordinates). The question is: how many components should we keep?
How many components? Drag the Kaiser line
Each bar is a component’s eigenvalue — the variance it explains. The Kaiser rule says: keep every component whose eigenvalue is greater than 1, i.e. every component that explains more than a single original variable would. Drag the threshold and watch how many survive.
Leave the line at 1 and four components survive — and those four already carry about 88% of the variance in eight features. Push the threshold up and you keep fewer (more compression, more information lost); pull it down and you keep more. The Kaiser cut at 1 is the common default because a component below it explains less than one of your original variables did, so it’s rarely earning its place.
Baseline vs PC regression
When we compare the two models, something instructive happens. The baseline model has a slightly higher adjusted R² — no surprise, it uses all the original information. But the PC model has a much higher F-statistic, because it reaches nearly the same explanatory power with far fewer predictors. It’s almost as informative and much more parsimonious.
And here’s the kicker. Check the variance inflation factors (our multicollinearity diagnostic from the regression chapters) and the PC model shows 1s straight down the line — no collinearity whatsoever. That shouldn’t surprise you: principal components are orthogonal, and orthogonal predictors cannot be collinear.
See it in code
The full pipeline — standardise, decompose, keep the Kaiser components, regress on them:
[coeff, score, latent] = pca(zscore(X)); % latent = eigenvalues
keep = latent > 1; % Kaiser criterion
scores = score(:, keep);
% then compare: fitlm(scores, y) vs fitlm(X, y)Static reference — the R and Python tabs run live in the page.
TipCheck your understanding
The PC model has a slightly lower adjusted R² than the baseline but a much higher F. Is that a problem?
No — it’s the whole point. The PC model reaches almost the same explanatory power with far fewer predictors, so it’s more parsimonious; the higher F reflects that efficiency. And because the components are orthogonal, it has zero multicollinearity (VIFs of 1). You trade a sliver of fit for a much cleaner, more stable model.
PCA is unsupervised — it finds directions of maximum variance in the predictors, with no knowledge of your outcome. A high-variance component isn’t guaranteed to predict y, and a component you discarded for low variance might have mattered. Don’t assume “keep the top components” is always optimal for prediction.
Where this shows up next
This closes the loop from Why Reduce Dimensions? and Eigenvectors & Eigenvalues. See Chapter (Dimension Reduction) for the full worked comparison, including the VIF table.