Momentum Optimizer

Accelerating Learning in the Right Direction

In the previous chapter, we studied why basic optimization methods such as Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-Batch Gradient Descent can struggle when training deep neural networks.

Although these methods update parameters by following the negative-gradient direction, they often suffer from two major problems:

• Slow convergence in flat directions
• Excessive oscillation in steep directions

These problems become particularly severe when the loss surface contains ravines—narrow valleys where the loss changes rapidly in one direction but slowly in another.

To overcome these limitations, optimization algorithms introduced an idea inspired by physics:

What if parameter updates could remember previous movement?

This idea leads to one of the most important improvements in optimization: Momentum.

Momentum allows an optimizer to accumulate velocity from previous updates. This enables faster movement in consistent directions while reducing unwanted oscillations.

In this chapter, we will understand the mathematical foundation, intuition, and optimization behavior of Momentum.

Why Standard Gradient Descent Is Slow

Recall the update rule for standard gradient descent:

where:

• $\mathbf{w}_t$ is the current parameter vector.
• $\alpha$ is the learning rate.
• $E(\mathbf{w})$ is the loss function.
• $\nabla_{\mathbf{w}}E(\mathbf{w}_t)$ is the gradient at iteration .

This update depends only on the current gradient.

The optimizer has no memory of previous directions.

As a result:

• If gradients fluctuate, updates become noisy.
• Inside ravines, the optimizer can oscillate from side to side.
• When gradients consistently point in one direction, the optimizer does not explicitly accumulate that consistency.

In other words:

Basic gradient descent behaves like a person taking independent steps without remembering previous movement.

The Physical Intuition Behind Momentum

Imagine pushing a heavy ball down a hill.

At the beginning, the ball moves slowly.

As it continues moving in the same direction, it accumulates speed.

Even if the slope temporarily becomes flatter, the ball continues moving because it has gained momentum.

This is the intuition behind Momentum optimization.

Instead of updating weights using only the current gradient, Momentum combines the current gradient with information from previous updates.

• Recent gradients contribute strongly.
• Older gradients gradually fade.
• Consistent directions accumulate.
• Alternating directions partially cancel.

The optimizer behaves like a moving object with velocity.

The Core Idea

Suppose gradients across several mini-batches keep pointing in roughly the same direction.

This suggests that the optimizer is following a reliable descent direction.

Instead of taking the same isolated step repeatedly, it can gradually build velocity.

Momentum follows two simple principles:

If gradients agree, accelerate.

If gradients fluctuate, smooth the movement.

Momentum Update Equations

Momentum introduces a variable called velocity.

Instead of updating the parameters directly from the current gradient, we first update the velocity.

Step 1: Update the Velocity

where: