Bias-Variance Tradeoff

Question

One of the most common opening questions in machine learning interviews:

Explain the bias-variance tradeoff. Decompose the expected prediction error of a model into its three components, explain what each component means, and describe how model complexity affects each of them. Finally, give an example of a high-bias model and a high-variance model.

Accepted Answer

Before deriving anything, let's set up the problem properly. Assume there is a true relationship between the features $x$ and the target $y$: \begin{equation} y = f(x) + \epsilon \end{equation} where $\epsilon$ is random noise with $E[\epsilon] = 0$ and $Var(\epsilon) = \sigma^2$. We train a model $\hat{f}$ on a training set, and the key insight is that $\hat{f}$ is itself random: if we had drawn a different training set, we would have obtained a (slightly) different model. So when we talk about expectations below, we average over two sources of randomness: the noise $\epsilon$ in a new observation, and the randomness of the training set that produced $\hat{f}$.

We want to decompose the expected squared prediction error at a point $x$: \begin{equation} E[(y - \hat{f}(x))^2] \end{equation} Step 1: separate the noiseSubstitute $y = f(x) + \epsilon$ (we drop the argument $x$ for readability): \begin{equation} E[(f + \epsilon - \hat{f})^2] = E[(f - \hat{f})^2] + 2E[\epsilon (f - \hat{f})] + E[\epsilon^2] \end{equation} Now look at the middle term. The noise $\epsilon$ belongs to the new observation, so it is independent of the trained model $\hat{f}$. For independent variables, the expectation of the product is the product of the expectations: \begin{equation} E[\epsilon (f - \hat{f})] = E[\epsilon] \cdot E[f - \hat{f}] = 0 \cdot E[f - \hat{f}] = 0 \end{equation} And since $E[\epsilon] = 0$, we have $E[\epsilon^2] = Var(\epsilon) = \sigma^2$. So far: \begin{equation} E[(y - \hat{f})^2] = E[(f - \hat{f})^2] + \sigma^2 \end{equation} Step 2: add and subtract the average modelLet $\bar{f} = E[\hat{f}]$ denote the average prediction over all possible training sets. We insert it into the first term: \begin{equation} f - \hat{f} = (f - \bar{f}) + (\bar{f} - \hat{f}) \end{equation} Squaring and taking expectations: \begin{equation} E[(f - \hat{f})^2] = (f - \bar{f})^2 + 2(f - \bar{f})E[\bar{f} - \hat{f}] + E[(\bar{f} - \hat{f})^2] \end{equation} Note that $(f - \bar{f})$ is a fixed number (not random), so it can be taken outside the expectation in the middle term. That middle term then vanishes because \begin{equation} E[\bar{f} - \hat{f}] = \bar{f} - E[\hat{f}] = \bar{f} - \bar{f} = 0 \end{equation} Putting everything together gives the famous decomposition: \begin{equation} E[(y - \hat{f})^2] = \underbrace{(f - E[\hat{f}])^2}_{	ext{Bias}^2} + \underbrace{E[(\hat{f} - E[\hat{f}])^2]}_{	ext{Variance}} + \underbrace{\sigma^2}_{	ext{Irreducible noise}} \end{equation} The three components mean the following:

 	Biashow far the average model is from the truth. High bias means the model is systematically wrong, no matter how much data you feed it. This is underfitting.
 	Variancehow much the model's predictions wiggle around their own average when you retrain on different samples. High variance means the model fits the noise of its particular training set. This is overfitting.
 	Irreducible noise $\sigma^2$: randomness in $y$ itself that no model can ever remove.

Now the tradeoff. As we increase model complexity (more features, deeper trees, smaller regularization), the bias term goes down — a flexible model can represent the truth more closely — but the variance term goes up, because a flexible model also has the freedom to chase noise. The training error keeps decreasing with complexity, but the test error follows a U-shape: it first drops (bias falling) and then rises again (variance exploding). The sweet spot is the bottom of the U, and we typically locate it with cross-validation.

Examples:

 	High biasfitting a straight-line regression to a clearly non-linear relationship. Even with infinite data the line cannot capture the curvature.
 	High variancea k-nearest-neighbours model with $k=1$, or a fully grown decision tree. Both can memorise the training data perfectly, so tiny changes in the training set produce completely different predictions.

Typical tools to manage the tradeoff are regularization (accept a little bias to remove a lot of variance), ensembling/bagging (averaging models lowers variance), and adding more training data (lowers variance, doesn't help bias).

Title	Category	Subcategory	Difficulty	Status
The Curse of Dimensionality	Machine Learning	Model Fundamentals	Medium	Unlock
Why a Tiny R-Squared Makes Money	Machine Learning	Model Fundamentals	Easy	Unlock

The Interview Guide for Quants and Traders

Bias-Variance Tradeoff

What do you think of this page?