Why Reduce Dimensions?

Info sheet · Statistics for Psychology & Neuroscience

Author

Andrew Bell

Published

August 13, 2026

Info sheet 0.2 (draft) · Prerequisites: correlation and variance · Give feedback ↗

Working notes for the author — not shown to students once collapsed; remove before publishing.

  • [ ]

What you’ll get from this sheet

Why would we ever throw away dimensions? Because when variables are correlated, many of them carry the same information twice — and a smaller set can hold most of it. By the end you should be able to:

  1. Give a few real situations where dimensionality reduction earns its keep.
  2. Explain why correlated variables are redundant, and how that lets us compress them.

Correlated variables don’t carry independent information — they overlap. Dimensionality reduction squeezes a large set of variables into a smaller set of new ones, throwing away as little useful information as possible.

Three scenarios

Let’s set the stage with three scenarios, each a little further from home than the last.

Scenario 1 — too many predictors. Return, one final time, to the Decade of Dance dataset. Suppose that instead of predicting a song’s popularity from one or two features, we wanted to use all of them: speechiness, acousticness, duration, valence, liveness, energy, tempo, and more. Many of these are correlated with one another, and throwing them all into a regression gives us a bloated, collinear model. Wouldn’t it be nice to compress those ten-or-so features into a handful of new variables that carry most of the same information?

Scenario 2 — high-density recording. EEG technology has advanced dramatically. Where we once recorded from 64 channels, we now routinely record from 256 or more. That’s wonderful for spatial resolution, but neighbouring channels are highly redundant, and analysing hundreds of correlated time series is unwieldy. Dimensionality reduction lets us summarise that redundancy.

Scenario 3 — populations of neurons. In systems neuroscience, we record hundreds of neurons simultaneously. The “state” of the population at any moment is a point in a very high-dimensional space. Remarkably, that activity often lives on a much lower-dimensional surface — a handful of dimensions capture the lion’s share of what the population is doing. Finding those dimensions is exactly what techniques like PCA are for.

Redundancy, made visible

Here are two variables measured on the same people. Slide their correlation up and watch the cloud collapse toward a line — and watch how the spread piles onto a single axis. The two bars show the variance along each of the data’s natural axes (its two eigenvalues).

At low correlation the cloud is a round blob — both axes carry real, separate information, and you genuinely need two dimensions. Crank the correlation up and the blob squashes onto a line: nearly all the variance loads onto one axis (λ₁), while the second (λ₂) shrinks toward zero. That second variable has become almost pure repetition — you could describe each person with a single number and barely lose a thing. That is why we can reduce dimensions: real data are full of this redundancy.

All of these problems can be tackled with some form of dimensionality-reduction algorithm — a method that reduces a large set of variables to a smaller set while throwing away as little useful information as possible. PCA is by far the most common, but not the only one: you may also meet Independent Components Analysis (ICA), exploratory and confirmatory factor analysis, Linear Discriminant Analysis (LDA), t-SNE, and UMAP. We focus on PCA, because the intuition it gives you carries over to the rest.

See it in code

Build two correlated variables and ask how much of their variance really lives on the first component:

z1 = randn(50,1); z2 = randn(50,1);
x = z1;  y = 0.95*z1 + sqrt(1 - 0.95^2)*z2;   % highly correlated with x
[~, ~, ~, ~, explained] = pca([x y]);
explained    % percent of variance per component

Static reference — the R and Python tabs run live in the page.

You have 8 predictors, but the first two principal components explain 90% of the variance. How is that possible?

Because the 8 predictors are correlated — they don’t carry 8 independent directions of information. Much of what one variable tells you, others already told you. PCA gathers that shared variance into a couple of components, so a handful of new axes reconstruct most of the original spread.

Reducing dimensions always throws away some information — the skill is dropping only the components that are mostly noise. And the new variables can be harder to interpret than the originals, since each principal component is a blend of all your measured variables.

Where this shows up next

See the Eigenvectors & Eigenvalues sheet for the machinery that finds these axes, and PCA in Action for the full pipeline on real predictors. Chapter (Dimension Reduction) has the complete treatment.