Entropy, Cross-Entropy and KL Divergence

Question

Information theory underpins almost every loss function used in classification, so interviewers like to test whether you really understand it.

Define Shannon entropy, cross-entropy and the Kullback-Leibler (KL) divergence. Explain what each one measures, prove that the KL divergence is always non-negative, explain why it is asymmetric, and finish by explaining the punchline: why minimising cross-entropy loss is equivalent to minimising KL divergence and to maximum likelihood. Include a small worked binary example.

Accepted Answer

Let's build the three quantities from first principles, starting with a single distribution and then comparing two.

Shannon entropySuppose an event has probability $p$. Its surprise (or information content) is defined as $-\log p$: a certain event ($p=1$) carries zero surprise, and rare events carry a lot. The entropy of a distribution $p$ is the expected surprise: \begin{equation} H(p) = -\sum_{i} p_i \log p_i \end{equation} It measures the average uncertainty of the distribution. If you use base-2 logarithms, $H(p)$ is exactly the minimum average number of bits needed to encode samples drawn from $p$ using an optimal code (Shannon's source coding theorem). A fair coin has $H = -	frac{1}{2}\log_2	frac{1}{2} - 	frac{1}{2}\log_2	frac{1}{2} = 1$ bit; a biased coin is more predictable, so its entropy is lower.

Cross-entropyNow suppose the data really comes from $p$, but we encode it using a code designed for a different distribution $q$ (our model). The average code length we pay is the cross-entropy: \begin{equation} H(p, q) = -\sum_{i} p_i \log q_i \end{equation} We still average over the true frequencies $p_i$, but charge each event the surprise $-\log q_i$ that our model assigns it.

KL divergenceThe extra cost of using the wrong code is the difference between the two: \begin{equation} D_{KL}(p \,\|\, q) = H(p, q) - H(p) = \sum_{i} p_i \log \frac{p_i}{q_i} \end{equation} This is the KL divergence from $q$ to $p$: how many extra bits per symbol we waste by believing $q$ when the truth is $p$.

Why $D_{KL} \geq 0$ (Gibbs' inequality)The cleanest proof uses Jensen's inequality. The logarithm is concave, so for any random variable $Z$, $E[\log Z] \leq \log E[Z]$. Write the negative divergence and apply this: \begin{equation} -D_{KL}(p \,\|\, q) = \sum_i p_i \log \frac{q_i}{p_i} \leq \log \sum_i p_i \frac{q_i}{p_i} = \log \sum_i q_i = \log 1 = 0 \end{equation} Hence $D_{KL}(p\,\|\,q) \geq 0$, with equality if and only if $q_i = p_i$ for every $i$ (because the $\log$ inequality is strict unless $q_i/p_i$ is constant). A useful corollary: since $H(p,q) = H(p) + D_{KL}(p\,\|\,q)$, cross-entropy is always at least the entropy. You can never encode data more efficiently than its own optimal code.

AsymmetryKL divergence is not a distance, because in general \begin{equation} D_{KL}(p \,\|\, q) 
eq D_{KL}(q \,\|\, p) \end{equation} The roles of the two distributions differ: $D_{KL}(p\,\|\,q)$ weights the discrepancies by the true probabilities $p_i$, so it heavily penalises a model $q$ that assigns near-zero probability to an outcome that actually happens often (the term $p_i \log(p_i/q_i)$ blows up as $q_i 	o 0$). Reversing the arguments changes which mismatches are punished. This is why "forward" KL fitting is mean-seeking (the model spreads to cover all the mass of $p$) while "reverse" KL is mode-seeking (the model latches onto one mode). It also fails the triangle inequality, so it is a divergence, not a metric.

The punchlineWhen we train a classifier, the true label distribution $p$ for a given example is fixed (it is the data); only the model $q_	heta$ depends on our parameters $	heta$. Decompose: \begin{equation} H(p, q_	heta) = \underbrace{H(p)}_{	ext{constant in }	heta} + D_{KL}(p \,\|\, q_	heta) \end{equation} Because $H(p)$ does not depend on $	heta$, minimising cross-entropy is exactly minimising the KL divergence from the model to the data distribution. And since cross-entropy averaged over the dataset is, up to a sign and a constant, the average negative log-likelihood, this is also maximum likelihood estimation. Three views, one objective: information-theoretic (waste fewest bits), distributional (get closest to the data distribution in KL), and statistical (make the observed data most probable).

Worked binary exampleSuppose the true label for an example is class 1, so $p = (0, 1)$, and the model predicts $q = (0.2, 0.8)$. The entropy of a one-hot $p$ is $H(p) = 0$ (no uncertainty). The cross-entropy is \begin{equation} H(p,q) = -[\,0 \cdot \log 0.2 + 1 \cdot \log 0.8\,] = -\log 0.8 \approx 0.223 \end{equation} and $D_{KL}(p\,\|\,q) = 0.223 - 0 = 0.223$ nats. If instead the model were confident and correct, $q = (0.01, 0.99)$, the loss falls to $-\log 0.99 \approx 0.010$; if it were confidently wrong, $q = (0.99, 0.01)$, the loss explodes to $-\log 0.01 \approx 4.6$. This asymmetric, unbounded penalty for confident mistakes is precisely the behaviour we want from a classification loss.

Trading context: a model that outputs probabilities of an up-move is scored with cross-entropy (log loss), which rewards calibrated confidence and savagely punishes a model that is sure the market will rise just before it crashes. KL divergence also appears directly when monitoring feature drift: compare today's feature distribution $q$ against the training distribution $p$ via $D_{KL}$ to flag when the live regime has wandered away from what the model was trained on.

Title	Category	Subcategory	Difficulty	Status
From Likelihood to Loss	Machine Learning	Foundations	Easy	Unlock
Maximum Likelihood Estimation for a Gaussian	Machine Learning	Foundations	Medium	Unlock

The Interview Guide for Quants and Traders

Entropy, Cross-Entropy and KL Divergence

What do you think of this page?