Plain recurrent neural networks struggle to learn long-range dependencies in a sequence. Explain why, using the idea of backpropagation through time, and describe how an LSTM's gating mechanism fixes the problem.

A recurrent network processes a sequence with a shared update \begin{equation} h_t = \phi(W h_{t-1} + U x_t + b) \end{equation} and the same weights are reused at every timestep. To train it we use backpropagation through time (BPTT): we unroll the network across the sequence and backpropagate. The gradient of a loss at time $t$ with respect to a hidden state far back at time $k$ involves a product of Jacobians, one per step: \begin{equation} \frac{\partial h_t}{\partial h_k} = \prod_{j=k+1}^{t} \frac{\partial h_j}{\partial h_{j-1}} \end{equation} Why it vanishes (or explodes)Each factor is roughly the recurrent weight times the derivative of the activation. Multiplying many such factors together behaves like raising a number to a high power. If the typical factor magnitude is below 1, the product shrinks exponentially with the distance $t - k$ — the gradient that should teach the network about events many steps ago decays to nearly zero (vanishing gradient), so the early inputs effectively cannot influence learning. If the typical factor is above 1, the product blows up (exploding gradient), causing unstable updates. Vanishing is the harder problem: it does not crash training, it just silently prevents long-range learning. How the LSTM fixes itAn LSTM adds a separate cell state $c_t$ whose update is additive rather than a repeated nonlinear squashing: \begin{equation} c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \end{equation} The forget gate $f_t$, input gate $i_t$ and output gate $o_t$ are sigmoids that the network learns. When the forget gate stays near 1, the cell state passes through almost unchanged, so the gradient along the cell path is multiplied by numbers close to 1 instead of decaying — often called the "constant error carousel" or gradient highway. The gates let the model learn what to remember and for how long, so information (and gradient) can survive across long spans. GRUs achieve the same effect with fewer gates. For financial sequences (long, noisy, with weak long-range structure) this matters, but it is not a cure-all: gradient clipping still guards against the exploding side, and very long-range dependencies are now often handled by attention instead.

Complete Training For Your Trading Interview

Interview Questions Machine Learning The Vanishing Gradient Problem

Unlock

Sorry, you don't have access to this interview question. Sign up to practice with our wide range of interview questions or use the filters to find free examples!

Python scratchpad

# Run Python right here in your browser
def two_sum(nums, target):
    seen = {}
    for i, n in enumerate(nums):
        if target - n in seen:
            return [seen[target - n], i]
        seen[n] = i
    return []

print(two_sum([2, 7, 11, 15], 9))  # [0, 1]

Run Python in your browser

Write and execute code — numpy, pandas & scikit-learn included — right on the question page.

My Notes

Notes are saved in this browser only. Sign up to keep them in your account.

The Interview Guide for Quants and Traders

The Vanishing Gradient Problem

What do you think of this page?