Batch, Stochastic and Mini-Batch Gradient Descent

Question

Training models on high-frequency datasets with hundreds of millions of rows forces a choice about how much data to use per gradient step.

Contrast full-batch, stochastic, and mini-batch gradient descent in terms of computational cost and gradient noise, and explain why the noise in SGD can actually help. Discuss why learning-rate schedules are needed. Then derive momentum as an exponentially weighted moving average of past gradients, and give a short Python implementation of SGD with momentum.

Accepted Answer

Gradient descent minimises a loss $L(w)$ by repeatedly stepping against the gradient, $w \leftarrow w - \eta 
abla L(w)$. The only question is how much data we use to estimate that gradient at each step, and the answer drives a fundamental cost-versus-noise trade-off.

Three variants

 	Full-batch gradient descent computes the gradient over the entire dataset before each step. The gradient is exact (low noise) and the descent path is smooth, but every single step costs a full pass over the data. On a high-frequency dataset of hundreds of millions of rows, one step might take minutes, so this is hopelessly slow.
 	Stochastic gradient descent (SGD) uses a single randomly chosen example per step. Each step is extremely cheap, so we take vastly more steps per unit time, but each gradient is a very noisy estimate of the true gradient, so the path zig-zags.
 	Mini-batch gradient descent is the practical middle ground: estimate the gradient on a small random batch (say 32 to 1024 examples). The noise falls like $1/\sqrt{B}$ in the batch size $B$, while vectorised hardware (GPUs) processes a whole batch almost as fast as one example. This is what is used in practice essentially everywhere.


Why noise helpsIt is tempting to see SGD's noise purely as a defect, but it has a real benefit. Loss surfaces of complex models are riddled with saddle points and shallow local minima where the exact gradient is (near) zero — full-batch descent can stall there. The noise in a stochastic gradient acts like a random kick: it knocks the iterate off saddle points and out of poor minima, helping it reach better regions. The noise also has a mild regularising effect, biasing training towards flatter minima that tend to generalise better — valuable when the signal-to-noise ratio is low, as it always is in trading data.

Learning-rate schedulesA fixed learning rate is a compromise. Early in training you want a large $\eta$ to make rapid progress; late in training that same $\eta$ makes the noisy SGD iterate bounce around the minimum without settling, because the gradient noise does not vanish. The fix is to decay $\eta$ over time — step decay, $1/t$ decay, or cosine annealing — so steps shrink as you approach the optimum and the iterate can settle. A short warm-up (ramping $\eta$ up over the first few hundred steps) is also standard to avoid an unstable initial blow-up.

Deriving momentum as an EWMA of gradientsPlain SGD treats each noisy gradient independently and so wastes information: in a long narrow valley it oscillates across the steep walls while crawling along the floor. Momentum fixes this by accumulating a velocity that averages recent gradients. The update is \begin{equation} v_t = \beta v_{t-1} + (1 - \beta) g_t, \qquad w_t = w_{t-1} - \eta \, v_t \end{equation} where $g_t = 
abla L(w_{t-1})$ and $\beta \in [0, 1)$. To see that $v_t$ really is an exponentially weighted moving average, unroll the recursion: \begin{equation} v_t = (1-\beta) g_t + \beta (1-\beta) g_{t-1} + \beta^2 (1-\beta) g_{t-2} + \dots = (1-\beta) \sum_{k=0}^{t-1} \beta^{k} g_{t-k} \end{equation} Each past gradient appears with a weight $\beta^k$ that decays geometrically into the past, and the weights $(1-\beta)\sum_k \beta^k$ sum to one — the definition of an EWMA with effective memory of roughly $1/(1-\beta)$ steps ($\beta = 0.9$ averages over about the last ten gradients). The benefit is twofold: noise that points in random directions averages out, while a consistent downhill direction reinforces step after step, so momentum damps the across-valley oscillation and accelerates along the valley floor.

Python: SGD with momentum
import numpy as np

def sgd_momentum(grad_fn, w0, X, y, lr=0.01, beta=0.9, batch=256, epochs=10):
    w = w0.copy()
    v = np.zeros_like(w)
    n = len(y)
    for _ in range(epochs):
        idx = np.random.permutation(n)          # reshuffle each epoch
        for start in range(0, n, batch):
            b = idx[start:start + batch]
            g = grad_fn(w, X[b], y[b])           # mini-batch gradient
            v = beta * v + (1 - beta) * g        # EWMA of gradients
            w = w - lr * v                       # step along velocity
    return w


Trading contextWhen training on high-frequency datasets too large to fit in memory, full-batch is a non-starter — you stream mini-batches from disk and take many cheap, noisy steps. The noise is doubly welcome here: it helps escape saddle points and nudges training towards flatter, better-generalising minima, which matters enormously when the predictive signal is faint and the data is mostly noise. Momentum and a decaying learning rate then make that noisy optimisation converge quickly and settle cleanly, and modern optimisers such as Adam simply combine this gradient EWMA with a second EWMA of squared gradients to adapt the step size per parameter.

Title	Category	Subcategory	Difficulty	Status
Gradient Descent From Scratch	Machine Learning	Optimization	Medium	Unlock
Newton's Method and Second-Order Optimization	Machine Learning	Optimization	Hard	Unlock

The Interview Guide for Quants and Traders

Batch, Stochastic and Mini-Batch Gradient Descent

What do you think of this page?