Backpropagation by Hand

Question

Consider the simplest possible neural network with one hidden layer. For input vector $x$:
\begin{equation} z = W_1 x + b_1, \qquad h = \sigma(z), \qquad \hat{y} = w_2^T h + b_2 \end{equation} with sigmoid activation $\sigma$ and squared-error loss $L = \frac{1}{2} (\hat{y} - y)^2$.

Derive the gradients of $L$ with respect to all four parameters ($W_1$, $b_1$, $w_2$, $b_2$) using backpropagation, and explain why backpropagation is computationally efficient compared to computing each gradient independently.

Accepted Answer

Backpropagation is nothing more than the chain rule, applied backwards through the network in an order that lets us reuse intermediate results. Let's set up the notation: $x \in \mathbb{R}^d$ is the input, $W_1$ is the (m x d) hidden weight matrix, $b_1 \in \mathbb{R}^m$, $w_2 \in \mathbb{R}^m$, and $b_2$ is a scalar. The forward pass computes, in order: \begin{equation} z = W_1 x + b_1, \qquad h = \sigma(z), \qquad \hat{y} = w_2^T h + b_2, \qquad L = \frac{1}{2}(\hat{y} - y)^2 \end{equation} We now walk backwards through these four equations, computing the derivative of $L$ with respect to each intermediate quantity.

Step 1: the output errorDifferentiate the loss with respect to the prediction (chain rule on the square, the inner derivative of $(\hat{y}-y)$ with respect to $\hat{y}$ being 1): \begin{equation} \delta \; := \; \frac{\partial L}{\partial \hat{y}} = \hat{y} - y \end{equation} This scalar is the error signal that everything below will reuse.

Step 2: the output layer parametersThe prediction is $\hat{y} = w_2^T h + b_2 = \sum_{j} w_{2,j} h_j + b_2$. Differentiating with respect to one weight $w_{2,j}$ gives $h_j$, and with respect to $b_2$ gives 1. Chaining with $\delta$: \begin{equation} \frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_2} = \delta \, h, \qquad \frac{\partial L}{\partial b_2} = \delta \end{equation} Step 3: propagate the error to the hidden layerHow does $L$ change when a hidden activation $h_j$ changes? Through $\hat{y}$ only, and $\frac{\partial \hat{y}}{\partial h_j} = w_{2,j}$: \begin{equation} \frac{\partial L}{\partial h} = \delta \, w_2 \end{equation} Next, push through the sigmoid. Because $\sigma$ is applied elementwise, $h_j = \sigma(z_j)$ depends only on $z_j$, and using $\sigma'(z) = \sigma(z)(1-\sigma(z))$: \begin{equation} \delta_1 \; := \; \frac{\partial L}{\partial z} = \frac{\partial L}{\partial h} \odot \sigma'(z) = \big(\delta \, w_2\big) \odot h \odot (1 - h) \end{equation} where $\odot$ denotes elementwise multiplication, and we exploited that $\sigma'(z)$ can be written using the already-computed forward value $h$ — no re-evaluation needed.

Step 4: the hidden layer parametersThe pre-activation is $z_j = \sum_{i} W_{1,ji} x_i + b_{1,j}$, so $\frac{\partial z_j}{\partial W_{1,ji}} = x_i$ and $\frac{\partial z_j}{\partial b_{1,j}} = 1$. Since entry $W_{1,ji}$ influences the loss only through $z_j$: \begin{equation} \frac{\partial L}{\partial W_{1,ji}} = \delta_{1,j} \, x_i \qquad \Longleftrightarrow \qquad \frac{\partial L}{\partial W_1} = \delta_1 \, x^T \end{equation} \begin{equation} \frac{\partial L}{\partial b_1} = \delta_1 \end{equation} (The outer product $\delta_1 x^T$ has exactly the (m x d) shape of $W_1$ — a useful sanity check.) That completes all four gradients, and a gradient descent step updates each parameter against its gradient, exactly as in logistic regression.

Why is this efficient?Notice the structure of what we just did: we computed $\delta$ once, reused it for $w_2$, $b_2$ and for $\delta_1$; we computed $\delta_1$ once and reused it for both $W_1$ and $b_1$. Every parameter's gradient came from one cheap local multiplication with an error signal that was already available. In general, one backward pass costs the same order of operations as one forward pass, so we get all gradients for the price of roughly two forward passes — for a network with millions of parameters, that is the difference between feasible and impossible. Compare the naive alternatives: differentiating each parameter independently repeats all the shared chain-rule work, and numerical finite differences \begin{equation} \frac{\partial L}{\partial 	heta_k} \approx \frac{L(	heta_k + \varepsilon) - L(	heta_k)}{\varepsilon} \end{equation} need one full extra forward pass per parameter — a million forward passes for a million parameters, against backprop's single backward sweep. (Finite differences survive in practice only as a debugging tool: "gradient checking" an implementation on a few parameters.)

One closing observation interviewers like, visible directly in our Step 3: every layer we pass through multiplies the error signal by $\sigma'(z) = h(1-h)$, which is at most $\frac{1}{4}$ (its maximum at $h = \frac{1}{2}$). Stack many sigmoid layers and the gradient shrinks geometrically towards zero on its way back — the vanishing gradient problem — which is the historical reason deep networks switched to activations like ReLU, whose derivative is exactly 1 on the active half of its domain.

Title	Category	Subcategory	Difficulty	Status
Weight Initialization: Xavier and He	Machine Learning	Neural Networks	Medium	Unlock
Why Do We Need Non-Linear Activations?	Machine Learning	Neural Networks	Easy	Unlock

The Interview Guide for Quants and Traders

Backpropagation by Hand

What do you think of this page?