backpropagation

How Networks Learn from Mistakes

Modern neural networks often consist of multiple layers and a large number of parameters, sometimes reaching millions. The central challenge in training such networks is determining how to efficiently update these parameters so that the model’s predictions improve over time. This learning process is achieved through an algorithm known as backpropagation, which systematically computes gradients of the loss function with respect to each parameter in the network.

Why Backpropagation Is Needed

Before understanding backpropagation, let us recall how learning happens in machine learning.

The goal of training is to minimize a loss function by adjusting the model's parameters, such as its weights and biases.

A basic gradient-descent algorithm follows an iterative process:

1. Perform a forward pass to generate predictions.
2. Compute the loss by comparing the predictions with the actual targets.
3. Calculate how the loss changes with respect to each parameter.
4. Update the parameters in the direction that reduces the loss.
5. Repeat the process until the loss becomes sufficiently small.

For a weight $w$, the update rule is:

For the complete weight vector $\mathbf{w}$:

where:

• $\alpha$ is the learning rate.
• $E(\mathbf{w})$ is the loss function.
• $\nabla_{\mathbf{w}}E(\mathbf{w})$ is the gradient of the loss with respect to the weights.

For simple machine-learning models such as linear regression, computing these gradients is relatively straightforward because the relationship between the weights and the output is direct.

Deep learning, however, introduces a more difficult challenge.

A deep neural network may contain:

• Multiple hidden layers
• Thousands, millions, or billions of parameters
• Complex chains of mathematical operations

During forward propagation, input features travel through one layer after another until the network produces a prediction.

The prediction is then compared with the true target to calculate a single loss value.

At this point, an important question arises:

Which specific parameters in the network were responsible for the error?

The final loss depends on every parameter in the network, often through many intermediate calculations.

A weight in an early layer may affect the output only indirectly through several hidden layers. Calculating the contribution of every weight manually would be extremely difficult and inefficient.

This is where backpropagation becomes essential.

Backpropagation efficiently calculates gradients for all trainable parameters by moving backward from the output layer toward the input layer.

It relies on three main ideas:

The Chain Rule of Calculus

The chain rule decomposes complex derivatives into a sequence of simpler derivatives.

Gradient Computation

Gradients measure how sensitive the loss is to changes in each parameter.

Error Propagation

The error signal is passed backward through the network so that responsibility for the prediction error can be distributed across all layers.

Rather than calculating every gradient independently, backpropagation reuses intermediate computations from the network.

This makes training computationally feasible even for very large neural networks.

Backpropagation computes the gradients, while the optimizer uses those gradients to update the parameters.

Without backpropagation, training modern deep-learning models containing millions or billions of parameters would be prohibitively slow and computationally impractical.

Build Intuition of Chain Rule of Calculus

Consider we have a function  $g(f(x))$

Chain Rule of Calculus (scaler case):

Consider,

$z=f(x)=x^2$

$y=g(z)=e^z=e^{x^2}$

We can calculate $\frac{dy}{dz}$ and $\frac{dz}{dx}$ easily.

But how would you calculate $\frac{dy}{dx}$ here? We have to apply chain rule here.

Chain Rule of Calculus (vector case):

Here each $z_i$ is a function of $x = \begin{bmatrix} x_1 \\ x_2 \\ .\\ .\\ x_m \end{bmatrix}$ and is a function of . So how do we calculate for any ?

Now to understand it simply, earlier for scaler case we just had only one path from $x$ to $z$ to $y$ ($x \to z \to y$). Now for vector case here we have three paths from $x_i$ to (considering ). So we have to apply chain rule in all the three paths and take the summation.

Basically,

What Is Our Target Here?

We want to train the model, that means we came up with the neural network model structure but we want to find the optimized weight parameters here. Weight parameters are named as $w_{ji}^{(l)}$ and it means the weight from node $i$ from previous layer to node $j$ in the current layer and current layer is $l$ (starting from the first hidden layer).

How do we find the optimizer proper weights? We run Gradient Descent algorithm and weight parameters are updated in each iteration.

$w^t$ contains all the weight parameters $w_{ji}^{(l)}$.

So at each iteration, our target is to find $\frac{\partial E}{\partial w_{ji}^{(l)}}$ for all the weight parameters.

What is $E$ here? $E$ is the Error or Loss we found comparing the actual output $(y)$ and predicted output $(\hat{y})$.

Loss/Error Functions

There are various types of Loss/Error functions depending on the task we want to perform. Here are few examples..

Mean Square Error(MSE)

Used mostly in linear regression problems.

Root Mean Square Error(RMSE)

Just taking square root of Mean Square Error

Mean Absolute Error (MAE)

Sum of absolute differences between actual and predicted outputs

Binary Cross-Entropy

Used in binary classification

Categorical Cross-Entropy

Used in multi-class classification when outputs are represented as one-hot encoding format and there are $K$ classes

Sparse Categorical Cross-Entropy

Used in multi-class classification when outputs are represented with class indices $(1,2,.. K)$, $y$ is the correct class index and $\hat{y}_y$ is the predicted probability of correct class

Backward Propagation of Errors: Step-by-Step Derivation

Backpropagation calculates gradients by starting at the output layer and moving backward through the network one layer at a time.

Let us derive the equations for a network that uses:

• A Sigmoid activation function in the output layer
• Binary Cross-Entropy as the loss function

Step 1: Output-Layer Gradient

For binary classification, the Binary Cross-Entropy loss is:

Let the output activation be:

We first calculate the derivative of the loss with respect to the output activation:

Combining the two fractions gives:

This tells us how sensitive the loss is to a change in the output activation.

Step 2: Gradient Through the Activation Function

The output activation is calculated using the Sigmoid function:

The derivative of the Sigmoid function is:

Therefore:

Using the chain rule:

Substituting the derivatives:

The common terms cancel:

The output-layer error signal is commonly denoted by:

Therefore:

This elegant simplification occurs when Binary Cross-Entropy is paired with a Sigmoid output activation.

Step 3: Gradients of the Output-Layer Weights

The linear combination of the output neuron is:

Consider the weight:

This weight connects hidden neuron $1$ in layer $1$ to output neuron $1$ in layer $2$.

Differentiating the output pre-activation with respect to this weight gives:

Using the chain rule:

Therefore:

Substituting the output error signal:

The gradient is the output error multiplied by the activation received through that weight.

More generally:

Step 4: Gradients of Hidden-Layer Weights

To propagate the gradient into an earlier hidden layer, we continue applying the chain rule.

Consider the first-layer weight:

Suppose hidden neuron $1$ uses ReLU:

The derivative of ReLU is:

Therefore:

The complete path from the loss to the first-layer weight is:

We already know:

Because:

we obtain:

For the hidden-layer linear combination:

we have:

Substituting all the local derivatives gives:

This equation demonstrates how gradient information is scaled and propagated backward through the network.

Visualizing Error Accumulation Across Multiple Paths

An activation in an early hidden layer may connect to several neurons in the next layer.

Suppose the hidden activation:

connects to two neurons with pre-activations:

Both downstream paths influence the loss.

Therefore, the total gradient with respect to $a_1^{(1)}$ is the sum of the gradients arriving through both paths:

Consider the two destination neurons:

and:

Their local derivatives are:

and:

Using:

and:

the total accumulated gradient becomes:

The gradients from all downstream paths are added together.

This allows the network to determine how an early neuron's shared activation affected every connected neuron in the following layer.

The hidden neuron's error signal is then:

Therefore:

Updating Bias Parameters

Bias parameters are optimized through backpropagation in the same way as weights.

Consider a bias parameter in layer $3$:

Differentiating the linear combination with respect to the bias gives:

Using the chain rule:

Since:

we obtain:

Therefore:

Because the local derivative of the pre-activation with respect to its bias is always $1$, the bias gradient is equal to the neuron's error signal.

Unlike a weight gradient, it is not multiplied by an incoming activation.

Key Takeaways

• Backpropagation calculates the gradients required to train a neural network's parameters.

• The algorithm uses the chain rule to decompose complex derivatives into manageable local derivatives.

• For a Sigmoid output paired with Binary Cross-Entropy, the output error signal simplifies to:

  $$\delta^{(L)} = \hat{y}-y$$

• A weight gradient is the destination neuron's error signal multiplied by the activation entering through that weight.

• Gradients from multiple downstream paths are added together during the backward pass.

• A bias gradient equals the neuron's error signal because its local derivative is $$.

Paired with Gradient Descent, backpropagation provides the computational mechanism that allows neural networks to learn from their errors.