Epochs, Batches, and Iterations

The Operational Framework of Training

When training a neural network, terms such as sample, batch, iteration, and epoch appear constantly. They describe how data moves through the training loop, how memory is used, and how often model parameters are updated.

Although closely related, each term represents a different part of the training process.

Understanding a Sample

A sample is one independent observation from a dataset.

For a dataset containing $n = 10{,}000$ rows, each row represents one training sample.

In supervised learning, a sample consists of an input and its target:

where:

• $\mathbf{x}_i$ contains the input features.
• $\mathbf{y}_i$ is the ground-truth label or target.

During forward propagation, the network transforms the input into a prediction. The prediction is compared with the target to calculate the loss.

What Is an Epoch?

An epoch is one complete pass through the entire training dataset.

During one epoch:

• Every training sample is processed once.
• Each sample participates in forward propagation.
• The resulting losses contribute to backpropagation.
• The parameters are updated after each processed batch.

Why Are Multiple Epochs Needed?

Neural networks usually begin with randomly initialized weights, so their first predictions are poor.

Each additional epoch gives the network another opportunity to:

• Evaluate its predictions.
• Calculate the loss.
• Compute gradients through backpropagation.
• Update its parameters.
• Gradually reduce the loss.

The number of epochs is therefore a training hyperparameter that controls how many times the model studies the complete dataset.

What Is a Batch?

Large datasets may contain millions of samples. Processing the entire dataset simultaneously can exceed available CPU or GPU memory.

To avoid this, the dataset is divided into smaller groups called batches.

A batch is a group of samples processed together during one forward and backward pass. The number of samples in each group is called the batch size, denoted by $m$.

Common batch sizes include:

Powers of two are common because they often work efficiently with modern hardware, although they are not mandatory.

Batch size affects:

• Memory usage
• Training speed
• GPU utilization
• Gradient noise
• Model generalization

Gradient Descent Variants

The batch size determines which form of Gradient Descent is being used.

A. Batch Gradient Descent

Batch Gradient Descent uses the entire dataset as one batch:

For $n = 10{,}000$ samples:

The model performs only one parameter update per epoch.

Advantages:

• Stable gradient estimates
• Smooth optimization path
• Deterministic updates under fixed conditions

Limitations:

• High memory usage
• Slow updates on large datasets
• Unsuitable for streaming data

B. Stochastic Gradient Descent

Pure Stochastic Gradient Descent uses one sample per batch:

For $n = 10{,}000$ samples:

The model performs 10,000 parameter updates during one epoch.

Advantages:

• Supports online learning
• Requires very little memory per update
• Noisy updates can help escape shallow minima

Limitations:

• Highly noisy gradients
• Erratic optimization path
• Inefficient GPU utilization

C. Mini-Batch Gradient Descent

Mini-Batch Gradient Descent divides the dataset into small groups and is the standard approach for modern deep learning.

Suppose:

Then:

The model performs 100 parameter updates per epoch.

Mini-batches provide a practical balance between:

• Gradient stability
• Training speed
• Memory efficiency
• GPU parallelism
• Useful optimization noise

What Is an Iteration?

An iteration is one complete parameter-update step.

During one iteration, the model:

1. Processes one batch through forward propagation.
2. Calculates the batch loss.
3. Computes gradients through backpropagation.
4. Updates its weights and biases.

Therefore:

One batch processed = One iteration = One parameter update

When the dataset size is exactly divisible by the batch size:

If it is not exactly divisible and the final smaller batch is retained:

Calculating Total Iterations

If training runs for $L$ epochs, the total number of parameter updates is:

When $n$ is divisible by $m$, this simplifies to:

Example

Suppose:

• Dataset size: $n = 10{,}000$
• Batch size: $m = 100$
• Epochs: $L = 5$

First, calculate the iterations per epoch:

Then calculate the total iterations:

The optimizer performs 500 parameter updates during the complete training run.

The Data-Flow Hierarchy

The training hierarchy can be summarized as follows:

1. Samples are grouped into batches.
2. Processing one batch completes one iteration.
3. Processing every batch once completes one epoch.
4. Repeating multiple epochs completes the training run.

Why Shuffle the Data?

Training data is usually shuffled before each epoch.

Shuffling changes which samples appear together in each mini-batch and provides several benefits:

• Breaks order bias: Prevents the model from learning accidental patterns caused by dataset ordering.
• Improves batch diversity: Makes each mini-batch more representative of the complete dataset.
• Reduces cyclic behavior: Avoids repeatedly presenting samples in the same sequence.
• Improves generalization: Encourages the model to learn meaningful patterns rather than ordering artifacts.

For sequence-dependent tasks such as time-series forecasting, shuffling must be used carefully because sample order may contain important information.

Key Takeaways

• A sample is one observation from the dataset.
• A batch is a group of samples processed together.
• An iteration processes one batch and performs one parameter update.
• An epoch completes one pass through the entire training dataset.
• Mini-Batch Gradient Descent balances speed, memory usage, and gradient stability.
• Shuffling changes batch composition and helps reduce ordering bias.

The core relationship is: