Deriving OLS

Question

Consider the linear model $y = X\beta + \epsilon$, where $X$ is the (n x p) feature matrix.

Derive the ordinary least squares estimator $\hat{\beta}$ in matrix form. Then show that it is unbiased, state the Gauss-Markov assumptions under which it is the best linear unbiased estimator, and explain what goes wrong when two features are highly correlated.

Accepted Answer

Before solving, we need two matrix calculus rules that play the role of our toolbox (both are the matrix analogues of $\frac{d}{dx} ax = a$ and $\frac{d}{dx} ax^2 = 2ax$):

 	For a constant vector $a$: \begin{equation} \frac{\partial \, a^T \beta}{\partial \beta} = a \end{equation}
 	For a symmetric matrix $A$: \begin{equation} \frac{\partial \, \beta^T A \beta}{\partial \beta} = 2A\beta \end{equation}

Step 1: write down the objectiveOLS chooses $\beta$ to minimise the residual sum of squares, which in matrix notation is \begin{equation} RSS(\beta) = (y - X\beta)^T (y - X\beta) \end{equation} Expanding the product (note that $y^T X \beta$ is a scalar, so it equals its own transpose $\beta^T X^T y$): \begin{equation} RSS(\beta) = y^T y - y^T X \beta - \beta^T X^T y + \beta^T X^T X \beta = y^T y - 2\beta^T X^T y + \beta^T X^T X \beta \end{equation} Step 2: differentiate and set to zeroApply the two toolbox rules ($X^T X$ is symmetric): \begin{equation} \frac{\partial RSS}{\partial \beta} = -2 X^T y + 2 X^T X \beta \end{equation} Setting this to zero gives the normal equations: \begin{equation} X^T X \hat{\beta} = X^T y \end{equation} and provided $X^T X$ is invertible: \begin{equation} \hat{\beta} = (X^T X)^{-1} X^T y \end{equation} This is a minimum (not a maximum) because the second derivative $2X^T X$ is positive semi-definite: for any vector $v$, $v^T X^T X v = \lVert Xv \rVert^2 \geq 0$, so the objective is a convex bowl.

Step 3: unbiasednessSubstitute the true model $y = X\beta + \epsilon$ into the estimator: \begin{equation} \hat{\beta} = (X^T X)^{-1} X^T (X\beta + \epsilon) = \beta + (X^T X)^{-1} X^T \epsilon \end{equation} where we used $(X^T X)^{-1} X^T X = I$. Taking expectations conditional on $X$ and using the assumption $E[\epsilon | X] = 0$: \begin{equation} E[\hat{\beta}] = \beta + (X^T X)^{-1} X^T E[\epsilon | X] = \beta \end{equation} The estimator is unbiased. The same decomposition gives its covariance under homoskedastic errors ($Var(\epsilon) = \sigma^2 I$). Using $Var(Az) = A \, Var(z) \, A^T$ for a constant matrix $A$: \begin{equation} Var(\hat{\beta}) = (X^T X)^{-1} X^T \, \sigma^2 I \, X (X^T X)^{-1} = \sigma^2 (X^T X)^{-1} \end{equation} Step 4: the Gauss-Markov theoremUnder the following assumptions:

 	Linearitythe true relationship is $y = X\beta + \epsilon$.
 	Exogeneity$E[\epsilon | X] = 0$ — the errors carry no information about the features.
 	Homoskedasticity and no autocorrelation$Var(\epsilon | X) = \sigma^2 I$ — all errors have equal variance and are uncorrelated with each other.
 	No perfect multicollinearity$X$ has full column rank, so $X^T X$ is invertible.

the OLS estimator is BLUE: the Best (minimum variance) Linear Unbiased Estimator. Note what is not required: normality of the errors (only needed for exact small-sample inference) — and note that financial returns routinely violate homoskedasticity (volatility clustering), which is why robust standard errors and GLS-type corrections exist.

Step 5: what goes wrong with highly correlated featuresIf two columns of $X$ are perfectly correlated, $X^T X$ is singular and $\hat{\beta}$ is not even defined: the data cannot distinguish $\beta = (1, 1)$ from $\beta = (100, -98)$ if the two features always move together. If they are highly but not perfectly correlated, $X^T X$ is invertible but nearly singular: some of its eigenvalues are close to zero, so $(X^T X)^{-1}$ — and with it $Var(\hat{\beta}) = \sigma^2 (X^T X)^{-1}$ — explodes along those directions. The symptoms: enormous, wildly unstable coefficients with opposite signs that swing completely on small data changes, while the predictions can still look fine (only the attribution between the twins is unstable). The standard diagnostic is the variance inflation factor, \begin{equation} VIF_j = \frac{1}{1 - R_j^2} \end{equation} where $R_j^2$ is the R-squared from regressing feature $j$ on all other features. The standard remedies: drop or combine the correlated features, replace them with principal components, or use ridge regression — which replaces $(X^T X)^{-1}$ by $(X^T X + \lambda I)^{-1}$, lifting all eigenvalues by $\lambda$ away from zero and stabilising the solution at the price of a small bias.

Title	Category	Subcategory	Difficulty	Status
Multicollinearity and the Variance Inflation Factor	Machine Learning	Linear Models	Medium	Unlock
The Gauss-Markov Theorem	Machine Learning	Linear Models	Hard	Unlock

The Interview Guide for Quants and Traders

Deriving OLS

What do you think of this page?