Introduction: The Problem with “Plain” SGD
Stochastic Gradient Descent (SGD) is one of the most widely used optimisation methods in machine learning. It updates model parameters by moving them in the direction that reduces the loss. In theory, this sounds straightforward. In practice, plain SGD can be slow and unstable, especially for deep learning and high-dimensional problems. Training may oscillate, take too long to reach a good solution, or get stuck making tiny progress along certain directions.
SGD Momentum was introduced to address these issues. The main idea is simple: instead of relying only on the current gradient, the update also uses a fraction of the previous update. This “memory” helps the optimiser build speed in consistent directions and reduce back-and-forth movement. For learners in a Data Scientist Course, momentum is a key concept because it explains why modern training runs converge faster than what basic SGD would achieve.
Understanding Momentum: Adding “Inertia” to Parameter Updates
In plain SGD, the parameter update at each step depends only on the current gradient. If the gradient direction changes frequently due to noise in mini-batches, updates can zig-zag. Momentum changes this by introducing a velocity term.
A practical way to think about it is “inertia”:
- If the optimiser keeps seeing gradients pointing in a similar direction, momentum accumulates speed, making bigger, more confident steps.
- If gradients fluctuate, momentum smooths the path, reducing erratic jumps.
Mechanically, momentum maintains a running vector (velocity) that is a combination of:
- the previous velocity, scaled by a momentum factor, and
- the current gradient, scaled by the learning rate.
Then the parameters are updated using this velocity rather than the raw gradient alone. This is why momentum is often described as “adding a fraction of the previous update vector to the current update.”
Why Momentum Accelerates Convergence
Momentum speeds up learning for two common reasons seen in real optimisation landscapes:
1) It Reduces Zig-Zagging in Narrow Valleys
Many loss surfaces have “ravines”: steep curvature in one direction and shallow curvature in another. Plain SGD may bounce across the steep direction while slowly moving along the shallow direction. This wastes steps. Momentum dampens oscillations across the steep dimension and builds speed along the shallow dimension, moving more directly toward the optimum.
2) It Helps Move Through Noisy Gradients
With mini-batch training, gradients are noisy estimates of the true gradient. One batch might suggest a slightly different direction than the next. Momentum averages these directional hints over time, producing smoother progress. This is particularly useful when training deep neural networks, where gradient noise and sharp curvature can appear together.
If you are taking a Data Science Course in Hyderabad and working on neural network training exercises, you will typically notice that adding momentum often reduces training time to reach a comparable loss, especially on messy datasets.
Key Hyperparameters: Learning Rate and Momentum Coefficient
Momentum introduces one main hyperparameter: the momentum coefficient, usually denoted as β (beta) or simply “momentum.” Typical values range from 0.8 to 0.99, with 0.9 being a common default.
Here is how the key settings interact:
- Learning rate (α): controls the step size. Too high can cause divergence; too low can slow training.
- Momentum (β): controls how much past updates influence the current direction. Higher momentum means stronger smoothing and more “carry-over.”
Practical guidance:
- If training oscillates heavily, slightly increasing momentum or reducing the learning rate may help.
- If training is too slow and stable, a modest learning rate increase (with momentum) can accelerate progress.
- Very high momentum can sometimes overshoot minima, especially with high learning rates.
In most workflows, momentum is tuned alongside learning rate, not in isolation.
Momentum Variants: Classical Momentum vs Nesterov Momentum
Two popular forms exist:
Classical Momentum
This is the standard approach described above: compute the velocity from past velocity and current gradient, then update parameters.
Nesterov Accelerated Gradient (NAG)
Nesterov momentum modifies the process by looking ahead: it computes the gradient after taking a partial step in the direction of the current velocity. This often provides a more responsive correction, reducing overshooting and improving stability in some settings.
You do not need Nesterov momentum for every model, but it is a useful option when standard momentum feels slightly too “aggressive.”
When Momentum Helps Most (and When It Matters Less)
Momentum is most beneficial when:
- the loss surface has ravines or ill-conditioned curvature,
- mini-batch gradients are noisy,
- training involves deep networks or large parameter spaces,
- convergence with plain SGD is slow or unstable.
Momentum may matter less when:
- the problem is small and convex (simple linear regression with good scaling),
- you are already using adaptive optimisers like Adam (which has momentum-like behaviour built in via moving averages),
- the dataset is tiny and gradients are relatively consistent.
That said, many production pipelines still use SGD with momentum because it can generalise well and perform strongly with careful learning rate schedules.
Conclusion: Momentum Is a Simple Change with a Big Impact
SGD Momentum improves optimisation by adding a fraction of the previous update vector to the current update. This creates smoother, faster progress by reducing zig-zagging and helping the optimiser build speed in consistent directions. The result is often quicker convergence and more stable training, especially in high-dimensional and noisy settings.
For learners building strong fundamentals through a Data Scientist Course, momentum is an essential concept because it connects mathematical intuition to practical training outcomes. And for practitioners applying these ideas in projects from a Data Science Course in Hyderabad, understanding momentum helps you tune models more effectively and diagnose why training might be slow, unstable, or stuck.
ExcelR – Data Science, Data Analytics and Business Analyst Course Training in Hyderabad
Address: Cyber Towers, PHASE-2, 5th Floor, Quadrant-2, HITEC City, Hyderabad, Telangana 500081
Phone: 096321 56744
