Adagrad Optimizer
Adaptive Learning Rates for Every Parameter
In the previous chapter, we studied Momentum, where the optimizer improved learning by looking at the direction of past gradients.
Momentum asked an important question:
If gradients keep pointing in the same direction, can we move faster in that direction?
Adagrad asks a completely different question.
Instead of focusing on direction, Adagrad focuses on magnitude. It asks:
Should every parameter in a neural network use the same learning rate?
The answer is often no.
Different parameters learn at different speeds, and forcing all of them to use the same learning rate can make optimization inefficient.
Adagrad solves this by giving each parameter its own adaptive learning rate.
The Problem with Fixed Learning Rates
Recall the standard gradient descent update rule:
where
Here, every parameter uses the same learning rate, .
This creates a fundamental problem. Some parameters receive large gradients frequently. Others receive very small gradients or are updated only occasionally.
Using one global learning rate means:
- Some parameters learn too fast.
- Some parameters learn too slowly.
This becomes especially problematic in large neural networks.
The Sparse Parameter Problem
Before understanding Adagrad, we first need to understand an important issue that appears frequently in machine learning: sparse features.
A sparse feature is a feature that is zero most of the time and only occasionally takes non-zero values.
Sparse features are extremely common in machine learning. For example: