Maximum Likelihood Estimation for a Gaussian

Question

Maximum likelihood is the engine behind most of the estimators a quant uses, and the Gaussian is the cleanest place to see it work end to end.

Given $n$ independent samples from a Gaussian $N(\mu, \sigma^2)$, write down the likelihood and the log-likelihood, differentiate with respect to $\mu$ and $\sigma^2$, and solve for the maximum likelihood estimators. Show that the variance estimator is biased and explain Bessel's correction. Finally, connect maximum likelihood under Gaussian noise to ordinary least squares.

Accepted Answer

Maximum likelihood estimation (MLE) chooses the parameters that make the observed data most probable. Let's carry it out for the Gaussian.

Step 1: the likelihoodWe have $n$ i.i.d. samples $x_1, \dots, x_n$ from $N(\mu, \sigma^2)$. The density of one point is \begin{equation} f(x_i \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}ight) \end{equation} Because the samples are independent, the likelihood of the whole dataset is the product: \begin{equation} L(\mu, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}ight) \end{equation} Step 2: the log-likelihoodProducts are awkward to differentiate, so we take logs (the $\log$ is monotone, so it does not move the maximum): \begin{equation} \ell(\mu, \sigma^2) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2 \end{equation} Step 3: solve for $\mu$Differentiate with respect to $\mu$ and set to zero. Only the last term depends on $\mu$: \begin{equation} \frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^{n}(x_i - \mu) = 0 \implies \sum_{i=1}^{n} x_i = n\mu \end{equation} \begin{equation} \hat\mu = \frac{1}{n}\sum_{i=1}^{n} x_i \end{equation} The MLE of the mean is simply the sample mean.

Step 4: solve for $\sigma^2$Treat $\sigma^2$ as a single variable $v$ and differentiate: \begin{equation} \frac{\partial \ell}{\partial v} = -\frac{n}{2v} + \frac{1}{2v^2}\sum_{i=1}^{n}(x_i - \mu)^2 = 0 \end{equation} Multiply through by $2v^2$ and rearrange: \begin{equation} n v = \sum_{i=1}^{n}(x_i - \mu)^2 \implies \hat\sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \hat\mu)^2 \end{equation} where we substitute the $\hat\mu$ found above. The MLE of the variance is the mean squared deviation from the sample mean, dividing by $n$.

Step 5: the variance MLE is biasedTake the expectation of $\hat\sigma^2$. The key subtlety is that $\hat\mu$ is itself computed from the same data, so the deviations $x_i - \hat\mu$ are slightly smaller on average than the deviations from the true $\mu$ would be — the sample mean sits, by construction, in the "middle" of the very points we are measuring. Working through the algebra, \begin{equation} E[\hat\sigma^2] = \frac{n-1}{n}\sigma^2 \end{equation} so the MLE systematically underestimates the variance by a factor $(n-1)/n$. The intuition: estimating $\mu$ from the data uses up one "degree of freedom", leaving only $n-1$ independent residuals.

Bessel's correctionTo get an unbiased estimator we divide by $n-1$ instead of $n$: \begin{equation} s^2 = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \hat\mu)^2, \qquad E[s^2] = \sigma^2 \end{equation} This is Bessel's correction. Note the honest tension: the MLE is biased but has lower variance and lower mean squared error; $s^2$ is unbiased but slightly noisier. For large $n$ the distinction is negligible (the two differ by a factor $1 + 1/(n-1)$), but for small samples it matters.

Connection to ordinary least squaresSuppose the data is generated by a linear model with Gaussian noise: $y_i = x_i^T \beta + \epsilon_i$, with $\epsilon_i \sim N(0, \sigma^2)$. The log-likelihood of $\beta$ is \begin{equation} \ell(\beta) = 	ext{const} - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i - x_i^T\beta)^2 \end{equation} Maximising $\ell$ over $\beta$ means minimising $\sum (y_i - x_i^T\beta)^2$ — exactly the ordinary least squares objective. So OLS is the maximum likelihood estimator under the assumption of Gaussian, constant-variance noise. This also tells you when least squares is the wrong tool: if the noise has heavy tails (a Laplace distribution), the MLE minimises the sum of absolute errors instead, which is far more robust to outliers.

Trading contextEstimating volatility from returns is a textbook MLE: assume returns are roughly $N(0, \sigma^2)$ over a short window and the MLE of $\sigma$ is the root-mean-square return. Two practical caveats follow directly from the derivation. First, for daily volatility we usually fix $\mu = 0$ rather than estimating it, because the true drift is tiny relative to the noise and estimating it just adds error — and with $\mu$ fixed we divide by $n$, not $n-1$. Second, real returns have fatter tails than the Gaussian, so the Gaussian MLE underestimates tail risk; this is exactly why practitioners reach for GARCH and Student-t likelihoods, both of which are still fitted by maximum likelihood on the same principle derived here.

Title	Category	Subcategory	Difficulty	Status
Entropy, Cross-Entropy and KL Divergence	Machine Learning	Foundations	Medium	Unlock
From Likelihood to Loss	Machine Learning	Foundations	Easy	Unlock

The Interview Guide for Quants and Traders

Maximum Likelihood Estimation for a Gaussian

What do you think of this page?