Weight Initialization: Xavier and He

Question

Before a neural network can learn, its weights have to be set to something. A naive choice can stop training before it starts.

Explain why initialising all weights to zero (or to any single constant) fails. Then explain why weights that are too large or too small make the signal variance explode or vanish as it propagates through a deep network. Derive the variance-preservation argument that leads to Xavier/Glorot initialisation for tanh/sigmoid networks and He initialisation for ReLU networks, and give a short Python snippet.

Accepted Answer

Initialisation matters because a deep network is a long chain of multiplications: a signal entering the first layer is multiplied by weight matrix after weight matrix on the way to the output, and the gradient is multiplied by them again on the way back. If those matrices are scaled badly, the signal either blows up or decays to nothing before it reaches the other end. Good initialisation keeps the signal's variance roughly constant across layers.

Why zeros (or equal weights) fail: the symmetry problemSuppose every weight in a layer is initialised to the same value (zero being the obvious case). Then every neuron in that layer receives the same input combination, computes the same output, and — crucially — receives the same gradient during backpropagation. Identical gradients mean identical updates, so the neurons stay identical forever. The whole layer behaves like a single neuron no matter how many units it nominally has; the network can never use its capacity. We need to break symmetry, which means initialising the weights randomly so that different neurons start in different places and learn different features. (Biases are commonly initialised to zero — that is fine, because the random weights already break the symmetry.)

Why the scale matters: exploding and vanishing signalRandom is necessary but not sufficient — the size of the random weights is just as important. Consider the pre-activation of one neuron, $z = \sum_{i=1}^{n} w_i x_i$, where $n = n_{in}$ is the number of incoming connections (fan-in). Assume the inputs $x_i$ and weights $w_i$ are independent, zero-mean, with the weights drawn i.i.d. with variance $	ext{Var}(w)$. The variance of a sum of independent zero-mean products is the sum of the variances: \begin{equation} 	ext{Var}(z) = \sum_{i=1}^{n} 	ext{Var}(w_i x_i) = n \, 	ext{Var}(w) \, 	ext{Var}(x) \end{equation} Now read off the consequence. Each layer multiplies the variance of its input by the factor $n \, 	ext{Var}(w)$. If that factor exceeds 1, the variance grows geometrically layer by layer and the activations (and gradients) explode; if it is below 1, the variance shrinks geometrically and they vanish. In a 30-layer network even a factor of $1.5$ or $0.7$ per layer is catastrophic. To keep the signal stable we want the per-layer factor to equal 1: \begin{equation} n \, 	ext{Var}(w) = 1 \implies 	ext{Var}(w) = \frac{1}{n_{in}} \end{equation} Xavier / Glorot initialisationThe condition above keeps the forward signal stable. By the same argument run backwards, keeping the gradient variance stable wants $	ext{Var}(w) = 1/n_{out}$. Glorot and Bengio compromised between the two by using the average of the fan-in and fan-out: \begin{equation} 	ext{Var}(w) = \frac{2}{n_{in} + n_{out}} \end{equation} In practice you either draw weights from a normal distribution with this variance (so standard deviation $1/\sqrt{n_{in}}$ in the fan-in-only version, or $\sqrt{2/(n_{in}+n_{out})}$ in the averaged version) or from a uniform distribution scaled to match. This is the right choice for activations that are roughly linear and symmetric around zero near the origin — tanh and sigmoid.

He initialisationReLU breaks the assumption behind Xavier: it zeroes out the negative half of its inputs, so on average it kills half the variance — passing a zero-mean signal through ReLU roughly halves its variance. To compensate we need the weights to inject twice as much variance, giving He (Kaiming) initialisation: \begin{equation} 	ext{Var}(w) = \frac{2}{n_{in}} \end{equation} i.e. standard deviation $\sqrt{2/n_{in}}$. This factor of 2 is exactly the correction for ReLU's discarded half, and it is the default for modern ReLU networks. The rule of thumb: Xavier for tanh/sigmoid, He for ReLU.
import numpy as np

def xavier_normal(n_in, n_out):
    # for tanh / sigmoid layers
    std = np.sqrt(2.0 / (n_in + n_out))
    return np.random.randn(n_in, n_out) * std

def he_normal(n_in, n_out):
    # for ReLU layers
    std = np.sqrt(2.0 / n_in)
    return np.random.randn(n_in, n_out) * std

# Quick sanity check: variance of activations stays ~1 across a deep ReLU stack
x = np.random.randn(1000, 256)
for _ in range(50):
    W = he_normal(x.shape[1], 256)
    x = np.maximum(0.0, x @ W)   # ReLU
print(x.var())   # stays order 1, neither explodes nor vanishes

If you swap he_normal for plain np.random.randn (variance 1, far too large for fan-in 256) in that loop, the activations explode to enormous values within a few layers; scale them down by a constant that is slightly too small and they collapse to zero. Getting the variance factor exactly right is what lets gradients flow through deep networks, and it is the reason batch/layer normalisation was later introduced — they make the network far more forgiving of imperfect initialisation, but a sane initialisation is still the first line of defence.

Title	Category	Subcategory	Difficulty	Status
Backpropagation by Hand	Machine Learning	Neural Networks	Hard	Unlock
Why Do We Need Non-Linear Activations?	Machine Learning	Neural Networks	Easy	Unlock

The Interview Guide for Quants and Traders

Weight Initialization: Xavier and He

What do you think of this page?