Adam Optimizer

Combining Momentum and Adaptive Learning

In the previous chapters, we studied three important optimizers that gradually improved the limitations of standard gradient descent.

Momentum introduced the idea of remembering past gradients so that optimization could move faster in consistent directions while reducing oscillations.

Adagrad introduced adaptive learning rates, allowing each parameter to learn at its own pace.

RMSProp improved Adagrad by preventing learning rates from shrinking indefinitely through exponentially weighted moving averages.

Each optimizer solved an important problem—but none solved everything.

This naturally leads to an important question:

Can we combine the strengths of Momentum and RMSProp into a single optimizer?

The answer is Adam.

Adam stands for:

Adaptive Moment Estimation

It is one of the most widely used optimization algorithms in deep learning because it combines the strongest ideas from previous optimizers.

Why Do We Need Adam?

Standard gradient descent uses the update rule:

where:

Its main problems include:

• Slow convergence
• Oscillatory movement in ravines
• The same learning-rate scale for every parameter

Momentum improves gradient descent by accumulating past gradients. It reduces oscillations and accelerates learning in consistent directions.

However, Momentum still uses a single global learning-rate scale.

Adagrad solves this by assigning adaptive learning rates to individual parameters. However, its accumulated squared-gradient history grows continuously, causing effective learning rates to decay too aggressively.

RMSProp solves this decay problem using an exponential moving average of squared gradients. However, RMSProp does not explicitly maintain a moving average of gradient directions for momentum-like acceleration.

Adam combines both ideas.

Core Idea Behind Adam

Adam combines two important mechanisms.

Momentum

Tracks the direction of recent gradients and smooths oscillations.

RMSProp

Tracks the magnitude of recent gradients and adapts the learning rate individually for each parameter.

In simple words:

Momentum asks:

Which direction should we move?

RMSProp asks:

How large should the step be for each parameter?

Adam answers both questions simultaneously.

Two Moving Averages

Adam maintains two separate exponential moving averages.

The first moving average tracks the gradients themselves. This behaves like Momentum.

The second moving average tracks squared gradients. This behaves like RMSProp.

Therefore, Adam tracks:

• Gradient direction
• Gradient magnitude

at the same time.

In this chapter, we use:

• $\mathbf{v}_t$ for the first-moment estimate of gradients
• $\mathbf{r}_t$ for the second-moment estimate of squared gradients

Step 1: Momentum Estimate

Adam first computes an exponentially weighted moving average of gradients:

where:

This equation behaves like normalized Momentum.

The variable $\mathbf{v}_t$ stores information about recent gradient directions. Its purpose is to:

• Smooth noisy updates
• Reduce oscillations
• Reinforce consistent directions

Step 2: RMSProp Estimate

Next, Adam computes an exponential moving average of squared gradients:

where:

This equation behaves like RMSProp.

The variable $\mathbf{r}_t$ stores information about recent gradient magnitudes. Its purpose is to create adaptive learning rates for individual parameters.

Step 3: Bias Correction

Adam usually initializes both moving averages to zero:

During the first few iterations, these moving averages are biased toward zero because they have not yet accumulated enough history.

Adam corrects this initialization bias using:

and:

The corrections matter most near the beginning of training. As $t$ increases, $\beta^{t+1}$ and $\rho^{t+1}$ approach zero, so the correction factors approach one.

Step 4: Weight Update

Finally, Adam updates the parameters using the bias-corrected estimates:

where $\delta$ is a small constant added for numerical stability.

Notice something important.

Unlike ordinary gradient descent, Adam does not directly use only the current gradient in the final update.

Instead:

• $\hat{\mathbf{v}}_{t+1}$ determines the smoothed direction.
• $\sqrt{\hat{\mathbf{r}}_{t+1}}$ controls the parameter-wise step scale.

Adam therefore separates two decisions:

Where should we move?

How aggressively should each parameter move?

The Complete Adam Algorithm

At each iteration, Adam performs the following calculations:

1. Compute the Gradient

2. Update the First Moment (Momentum estimate)

3. Update the Second Moment (RMSProp Estimate)

4. Correct the Initialization Bias

5. Update the Parameters

Understanding the Components

The first-moment estimate $\mathbf{v}_{t+1}$ tracks the average direction of recent gradients.

It reduces oscillations and supports faster movement in stable directions.

The second-moment estimate $\mathbf{r}_{t+1}$ tracks the recent magnitude of gradients.

• Large gradients produce larger values in $\mathbf{r}_{t+1}$, reducing the corresponding effective learning rates.
• Small or infrequent gradients keep their components in $\mathbf{r}_{t+1}$ relatively small, allowing comparatively larger learning rates.

Thus, Adam learns both direction and scale.

Why Adaptive Learning Matters

Suppose some parameters repeatedly receive large gradients.

Their components in $\mathbf{r}_{t+1}$ become large. As a result:

becomes smaller for those parameters.

They automatically slow down.

Now suppose another parameter receives updates only occasionally.

Its second-moment estimate remains relatively small, so its effective learning rate stays comparatively large.

This allows sparse parameters to continue learning efficiently.

Why Momentum Matters

Consider optimization inside a ravine.

Without Momentum, gradients frequently change direction across the steep walls. The optimizer repeatedly oscillates sideways.

The first-moment estimate accumulates gradients over time:

• Consistent directions reinforce one another.
• Opposite directions partially cancel.
• Oscillation decreases.
• Movement becomes smoother.
• Optimization progresses faster along stable directions.

Adam inherits these benefits through $\mathbf{v}_t$.

Adam as a Combination of Previous Optimizers

Adam combines both sets of strengths, making it robust across many neural-network architectures and training problems.

Why Adam Works So Well

Adam became popular because it performs well in many situations without extensive tuning.

It works well because:

• It accelerates learning when gradients are consistent.
• It reduces oscillations in steep ravines.
• It adapts learning rates individually for each parameter.
• It avoids Adagrad's permanently growing accumulator.
• It handles sparse gradients effectively.
• Its default hyperparameters often provide a strong starting point.

These properties make Adam highly practical for large-scale deep learning.

Typical Hyperparameters

These values are common starting points, not universal guarantees. The best settings can still depend on the model, dataset, and training objective.

Intuition Behind Adam

A useful way to think about Adam is to imagine driving a car toward a destination.

Momentum acts like accumulated speed.

If you keep moving in the same direction, you gradually accelerate.

RMSProp acts like adaptive control over how aggressively each movement should be made.

When recent gradients are large, the corresponding step is reduced. When they are smaller or sparse, the parameter can take a relatively larger step.

Adam combines both mechanisms.

It accelerates in consistent directions and controls the step size using recent gradient magnitudes.

This creates stable and efficient optimization.

The Evolution of Optimizers

We can now see how optimization algorithms evolved:

1. Gradient Descent introduced iterative optimization.
2. Momentum introduced directional memory.
3. Adagrad introduced adaptive learning rates.
4. RMSProp introduced controlled forgetting.
5. Adam combined directional memory and adaptive scaling.

Each optimizer addressed limitations found in earlier methods.

This gradual evolution is one of the clearest examples of progress in deep learning.

Key Takeaways

Adam combines ideas from Momentum and RMSProp.

It tracks two moving averages:

• $\mathbf{v}_t$ tracks recent gradient direction.
• $\mathbf{r}_t$ tracks recent gradient magnitude.

It also applies bias correction to compensate for the zero initialization of these estimates.

Together, these mechanisms allow Adam to:

• Reduce oscillations
• Accelerate learning in stable directions
• Adapt the learning rate for each parameter
• Avoid Adagrad's indefinitely growing accumulator
• Handle sparse gradients effectively
• Remain practical across many deep-learning tasks

Final Insight

Adam combines several major ideas from optimization research into a single powerful algorithm.

At every step, Adam decides where to move and how aggressively each parameter should move, making it one of the most effective and widely used optimizers in deep learning.