Principal Component Analysis

Question

Derive Principal Component Analysis from first principles: show that the direction maximising the variance of the projected data is the eigenvector of the covariance matrix with the largest eigenvalue.

Then explain how PCA is applied to the yield curve, and what the first three components represent there.

Accepted Answer

PCA looks for the directions in feature space along which the data varies the most. Let's build the derivation up carefully.

Step 1: the variance of a projectionLet $x$ be a (centered) random feature vector with covariance matrix $\Sigma = E[x x^T]$, and let $w$ be a candidate direction. The projection of the data onto $w$ is the scalar $w^T x$, and its variance is \begin{equation} Var(w^T x) = E[(w^T x)^2] = E[w^T x \, x^T w] = w^T E[x x^T] w = w^T \Sigma w \end{equation} where we could pull $w$ outside the expectation because it is a constant vector. So "find the direction of maximal variance" becomes: \begin{equation} \max_{w} \; w^T \Sigma w \end{equation} Step 2: why we need a constraintAs stated, the problem is ill-posed: doubling $w$ multiplies $w^T \Sigma w$ by four, so we could make the "variance" arbitrarily large by scaling. We only care about the direction, so we fix the length: \begin{equation} \max_{w} \; w^T \Sigma w \quad 	ext{subject to} \quad w^T w = 1 \end{equation} Step 3: Lagrange multipliersTo maximise a function under an equality constraint, we form the Lagrangian — the objective minus a multiplier $\lambda$ times the constraint — and set its gradient to zero. The intuition: at a constrained optimum, the gradient of the objective must be parallel to the gradient of the constraint (otherwise we could slide along the constraint surface and improve), and $\lambda$ is exactly that proportionality factor. \begin{equation} \mathcal{L}(w, \lambda) = w^T \Sigma w - \lambda (w^T w - 1) \end{equation} Using the rule $\frac{\partial}{\partial w} w^T A w = 2Aw$ for symmetric $A$ (both $\Sigma$ and $I$ are symmetric): \begin{equation} \frac{\partial \mathcal{L}}{\partial w} = 2 \Sigma w - 2 \lambda w = 0 \end{equation} \begin{equation} \Sigma w = \lambda w \end{equation} This is precisely the eigenvalue equation: any constrained stationary point $w$ must be an eigenvector of $\Sigma$, with the multiplier $\lambda$ as its eigenvalue.

Step 4: which eigenvector?Plug the condition back into the objective. For an eigenvector with $w^T w = 1$: \begin{equation} Var(w^T x) = w^T \Sigma w = w^T (\lambda w) = \lambda \, w^T w = \lambda \end{equation} The variance captured by an eigenvector is its eigenvalue. To maximise variance we therefore pick the eigenvector with the largest eigenvalue — that is the first principal component. The second principal component solves the same problem with the extra requirement of being orthogonal to the first, which (because a symmetric matrix has orthogonal eigenvectors) yields the eigenvector with the second-largest eigenvalue, and so on. The fraction of total variance explained by the first $k$ components is \begin{equation} \frac{\lambda_1 + ... + \lambda_k}{\lambda_1 + ... + \lambda_p} \end{equation} which is the standard criterion for choosing how many components to keep. Two practical notes: the data must be centered first (otherwise the "largest variance direction" just points at the mean), and if features live on different scales they should be standardised, since PCA otherwise loads on whichever feature has the biggest units.

Step 5: PCA on the yield curveTake daily changes of swap or bond yields at maturities of, say, 1, 2, 5, 10, 20 and 30 years — a 6-dimensional, heavily correlated dataset. Run PCA on its covariance matrix. The empirical result is remarkably stable across markets and decades, with the first three components typically explaining 95-99% of the variance:

PC1 — Level (often ~80-90% of variance)all maturities move with the same sign — a parallel shift of the whole curve up or down.
 	PC2 — Slopeshort maturities move opposite to long maturities — the curve steepens or flattens.
 	PC3 — Curvaturethe middle of the curve moves against both ends — the curve bows.

This is enormously useful in practice: a rates book with dozens of instruments can be risk-managed in terms of three interpretable factors — hedge the level exposure, then the slope, then the curvature — instead of dozens of individual yields. The same technique applies to any correlated asset universe: the first component of a stock universe is essentially "the market", and statistical arbitrage desks trade the residuals that remain after stripping out the dominant components. The caveat to mention in an interview: principal components are statistical constructs estimated from a window of data — the loadings can shift in stressed markets exactly when you rely on them, so factor stability needs monitoring.

Title	Category	Subcategory	Difficulty	Status
How Many Principal Components?	Machine Learning	Dimensionality Reduction	Medium	Unlock
t-SNE and UMAP: Nonlinear Embeddings	Machine Learning	Dimensionality Reduction	Medium	Unlock

The Interview Guide for Quants and Traders

Principal Component Analysis

What do you think of this page?