Course contentsShow

backpropagation

How Networks Learn from Mistakes

Modern neural networks often consist of multiple layers and a large number of parameters, sometimes reaching millions. The central challenge in training such networks is determining how to efficiently update these parameters so that the model’s predictions improve over time. This learning process is achieved through an algorithm known as backpropagation, which systematically computes gradients of the loss function with respect to each parameter in the network.

Why Backpropagation Is Needed

Before understanding backpropagation, let us recall how learning happens in machine learning.

The goal of training is to minimize a loss function by adjusting the model's parameters, such as its weights and biases.

A basic gradient-descent algorithm follows an iterative process:

Perform a forward pass to generate predictions.
Compute the loss by comparing the predictions with the actual targets.
Calculate how the loss changes with respect to each parameter.
Update the parameters in the direction that reduces the loss.
Repeat the process until the loss becomes sufficiently small.

For a weight $w$ , the update rule is:

w \leftarrow w-\alpha\frac{\partial E}{\partial w}

For the complete weight vector $\mathbf{w}$ :

\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_{\mathbf{w}}E(\mathbf{w})

where:

$\alpha$ is the learning rate.
$E(\mathbf{w})$ is the loss function.
$\nabla_{\mathbf{w}}E(\mathbf{w})$ is the gradient of the loss with respect to the weights.

For simple machine-learning models such as linear regression, computing these gradients is relatively straightforward because the relationship between the weights and the output is direct.

Deep learning, however, introduces a more difficult challenge.

A deep neural network may contain:

Multiple hidden layers
Thousands, millions, or billions of parameters
Complex chains of mathematical operations

During forward propagation, input features travel through one layer after another until the network produces a prediction.

The prediction is then compared with the true target to calculate a single loss value.

At this point, an important question arises:

Which specific parameters in the network were responsible for the error?

The final loss depends on every parameter in the network, often through many intermediate calculations.

A weight in an early layer may affect the output only indirectly through several hidden layers. Calculating the contribution of every weight manually would be extremely difficult and inefficient.

This is where backpropagation becomes essential.

Backpropagation efficiently calculates gradients for all trainable parameters by moving backward from the output layer toward the input layer.

It relies on three main ideas:

The Chain Rule of Calculus

The chain rule decomposes complex derivatives into a sequence of simpler derivatives.

Gradient Computation

Gradients measure how sensitive the loss is to changes in each parameter.

Error Propagation

The error signal is passed backward through the network so that responsibility for the prediction error can be distributed across all layers.

Rather than calculating every gradient independently, backpropagation reuses intermediate computations from the network.

This makes training computationally feasible even for very large neural networks.

Backpropagation computes the gradients, while the optimizer uses those gradients to update the parameters.

Without backpropagation, training modern deep-learning models containing millions or billions of parameters would be prohibitively slow and computationally impractical.

Build Intuition of Chain Rule of Calculus

Consider we have a function $g(f(x))$

Chain Rule of Calculus (scaler case):

Consider,

$z=f(x)=x^2$

$y=g(z)=e^z=e^{x^2}$

We can calculate $\frac{dy}{dz}$ and $\frac{dz}{dx}$ easily.

\frac{dy}{dz}=e^z, \;\;\;\;\; \frac{dz}{dx}=2x

But how would you calculate $\frac{dy}{dx}$ here? We have to apply chain rule here.

\frac{dy}{dx}=\frac{dy}{dz}.\frac{dz}{dx}

=e^z.2x=e^{x^2}.2x

Chain Rule of Calculus (vector case):

Here each $z_i$ is a function of $x = \begin{bmatrix} x_1 \\ x_2 \\ .\\ .\\ x_m \end{bmatrix}$ and $y$ is a function of $z = \begin{bmatrix} z_1 \\ z_2 \\ .\\ .\\ z_n \end{bmatrix}$ . So how do we calculate $\frac{\partial y}{\partial x_i}$ for any $x_i$ ?

Now to understand it simply, earlier for scaler case we just had only one path from $x$ to $z$ to $y$ ( $x \to z \to y$ ). Now for vector case here we have three paths from $x_i$ to $y$ (considering $n=3$ ). So we have to apply chain rule in all the three paths and take the summation.

\frac{\partial y}{\partial x_i}=\frac{\partial y}{\partial z_1}.\frac{\partial z_1}{\partial x_i}+ \frac{\partial y}{\partial z_2}.\frac{\partial z_2}{\partial x_i}+ \frac{\partial y}{\partial z_3}.\frac{\partial z_3}{\partial x_i}

Basically,

\frac{\partial y}{\partial x_i}=\sum_{j=1}^n \frac{\partial y}{\partial z_j}.\frac{\partial z_j}{\partial x_i}

What Is Our Target Here?

We want to train the model, that means we came up with the neural network model structure but we want to find the optimized weight parameters here. Weight parameters are named as $w_{ji}^{(l)}$ and it means the weight from node $i$ from previous layer to node $j$ in the current layer and current layer is $l$ (starting from the first hidden layer).

How do we find the optimizer proper weights? We run Gradient Descent algorithm and weight parameters are updated in each iteration.

w^{t+1} = w^t - \alpha \frac{\partial E}{\partial w^t}

$w^t$ contains all the weight parameters $w_{ji}^{(l)}$ .

So at each iteration, our target is to find $\frac{\partial E}{\partial w_{ji}^{(l)}}$ for all the weight parameters.

What is $E$ here? $E$ is the Error or Loss we found comparing the actual output $(y)$ and predicted output $(\hat{y})$ .

Loss/Error Functions

There are various types of Loss/Error functions depending on the task we want to perform. Here are few examples..

Mean Square Error(MSE)

Used mostly in linear regression problems.

E= \frac{1}{n} \sum_{i=1}^n {(y_i-\hat{y_i})}^2

Root Mean Square Error(RMSE)

Just taking square root of Mean Square Error

E= \sqrt{\frac{1}{n} \sum_{i=1}^n {(y_i-\hat{y_i})}^2}

Mean Absolute Error (MAE)

Sum of absolute differences between actual and predicted outputs

E= \frac{1}{n} \sum_{i=1}^n {|y_i-\hat{y_i}|}

Binary Cross-Entropy

Used in binary classification

E= -y \; log(\hat{y}) - (1-y)log (1-\hat{y})

Categorical Cross-Entropy

Used in multi-class classification when outputs are represented as one-hot encoding format and there are $K$ classes

E= - \sum_{k=1}^K y_k log(\hat{y_k})

Sparse Categorical Cross-Entropy

Used in multi-class classification when outputs are represented with class indices $(1,2,.. K)$ , $y$ is the correct class index and $\hat{y}_y$ is the predicted probability of correct class $y$

E = - log(\hat{y}_y)

Backward Propagation of Errors: Step-by-Step Derivation

Backpropagation calculates gradients by starting at the output layer and moving backward through the network one layer at a time.

Let us derive the equations for a network that uses:

A Sigmoid activation function in the output layer
Binary Cross-Entropy as the loss function

Step 1: Output-Layer Gradient

For binary classification, the Binary Cross-Entropy loss is:

E = -y\log(\hat{y}) - (1-y)\log(1-\hat{y})

Let the output activation be:

a_1^{(2)} = \hat{y}

We first calculate the derivative of the loss with respect to the output activation:

\frac{\partial E} {\partial a_1^{(2)}} = -\frac{y}{a_1^{(2)}} + \frac{1-y}{1-a_1^{(2)}}

Combining the two fractions gives:

\frac{\partial E} {\partial a_1^{(2)}} = \frac{a_1^{(2)}-y} {a_1^{(2)}\left(1-a_1^{(2)}\right)}

This tells us how sensitive the loss is to a change in the output activation.

Step 2: Gradient Through the Activation Function

The output activation is calculated using the Sigmoid function:

a_1^{(2)} = \sigma\left(z_1^{(2)}\right)

The derivative of the Sigmoid function is:

\sigma'(z) = \sigma(z)\left(1-\sigma(z)\right)

Therefore:

\frac{\partial a_1^{(2)}} {\partial z_1^{(2)}} = a_1^{(2)} \left( 1-a_1^{(2)} \right)

Using the chain rule:

\frac{\partial E} {\partial z_1^{(2)}} = \frac{\partial E} {\partial a_1^{(2)}} \cdot \frac{\partial a_1^{(2)}} {\partial z_1^{(2)}}

Substituting the derivatives:

\frac{\partial E} {\partial z_1^{(2)}} = \frac{a_1^{(2)}-y} {a_1^{(2)}\left(1-a_1^{(2)}\right)} \cdot a_1^{(2)} \left(1-a_1^{(2)}\right)

The common terms cancel:

\frac{\partial E} {\partial z_1^{(2)}} = a_1^{(2)}-y

The output-layer error signal is commonly denoted by:

\delta_1^{(2)} = \frac{\partial E} {\partial z_1^{(2)}}

Therefore:

\boxed{ \delta_1^{(2)} = a_1^{(2)}-y }

This elegant simplification occurs when Binary Cross-Entropy is paired with a Sigmoid output activation.

Step 3: Gradients of the Output-Layer Weights

The linear combination of the output neuron is:

z_1^{(2)} = w_{11}^{(2)}a_1^{(1)} + w_{12}^{(2)}a_2^{(1)} + w_{13}^{(2)}a_3^{(1)} + b_1^{(2)}

Consider the weight:

w_{11}^{(2)}

This weight connects hidden neuron $1$ in layer $1$ to output neuron $1$ in layer $2$ .

Differentiating the output pre-activation with respect to this weight gives:

\frac{\partial z_1^{(2)}} {\partial w_{11}^{(2)}} = a_1^{(1)}

Using the chain rule:

\frac{\partial E} {\partial w_{11}^{(2)}} = \frac{\partial E} {\partial z_1^{(2)}} \cdot \frac{\partial z_1^{(2)}} {\partial w_{11}^{(2)}}

Therefore:

\frac{\partial E} {\partial w_{11}^{(2)}} = \delta_1^{(2)} a_1^{(1)}

Substituting the output error signal:

\boxed{ \frac{\partial E} {\partial w_{11}^{(2)}} = \left( a_1^{(2)}-y \right) a_1^{(1)} }

The gradient is the output error multiplied by the activation received through that weight.

More generally:

\boxed{ \frac{\partial E} {\partial w_{1i}^{(2)}} = \delta_1^{(2)} a_i^{(1)} }

Step 4: Gradients of Hidden-Layer Weights

To propagate the gradient into an earlier hidden layer, we continue applying the chain rule.

Consider the first-layer weight:

w_{11}^{(1)}

Suppose hidden neuron $1$ uses ReLU:

a_1^{(1)} = \operatorname{ReLU} \left( z_1^{(1)} \right)

The derivative of ReLU is:

\operatorname{ReLU}'(z) = \begin{cases} 1, & z>0 \\ 0, & z\leq0 \end{cases}

Therefore:

\frac{\partial a_1^{(1)}} {\partial z_1^{(1)}} = \operatorname{ReLU}' \left( z_1^{(1)} \right)

The complete path from the loss to the first-layer weight is:

\frac{\partial E} {\partial w_{11}^{(1)}} = \frac{\partial E} {\partial z_1^{(2)}} \cdot \frac{\partial z_1^{(2)}} {\partial a_1^{(1)}} \cdot \frac{\partial a_1^{(1)}} {\partial z_1^{(1)}} \cdot \frac{\partial z_1^{(1)}} {\partial w_{11}^{(1)}}

We already know:

\frac{\partial E} {\partial z_1^{(2)}} = \delta_1^{(2)}

Because:

z_1^{(2)} = w_{11}^{(2)}a_1^{(1)} + w_{12}^{(2)}a_2^{(1)} + \cdots + b_1^{(2)}

we obtain:

\frac{\partial z_1^{(2)}} {\partial a_1^{(1)}} = w_{11}^{(2)}

For the hidden-layer linear combination:

z_1^{(1)} = w_{11}^{(1)}x_1 + w_{12}^{(1)}x_2 + \cdots + b_1^{(1)}

we have:

\frac{\partial z_1^{(1)}} {\partial w_{11}^{(1)}} = x_1

Substituting all the local derivatives gives:

\boxed{ \frac{\partial E} {\partial w_{11}^{(1)}} = \delta_1^{(2)} w_{11}^{(2)} \operatorname{ReLU}' \left( z_1^{(1)} \right) x_1 }

This equation demonstrates how gradient information is scaled and propagated backward through the network.

Visualizing Error Accumulation Across Multiple Paths

An activation in an early hidden layer may connect to several neurons in the next layer.

Suppose the hidden activation:

a_1^{(1)}

connects to two neurons with pre-activations:

z_1^{(2)} \qquad\text{and}\qquad z_2^{(2)}

Both downstream paths influence the loss.

Therefore, the total gradient with respect to $a_1^{(1)}$ is the sum of the gradients arriving through both paths:

\frac{\partial E} {\partial a_1^{(1)}} = \left( \frac{\partial E} {\partial z_1^{(2)}} \cdot \frac{\partial z_1^{(2)}} {\partial a_1^{(1)}} \right) + \left( \frac{\partial E} {\partial z_2^{(2)}} \cdot \frac{\partial z_2^{(2)}} {\partial a_1^{(1)}} \right)

Consider the two destination neurons:

z_1^{(2)} = w_{11}^{(2)}a_1^{(1)} + w_{12}^{(2)}a_2^{(1)} + \cdots

and:

z_2^{(2)} = w_{21}^{(2)}a_1^{(1)} + w_{22}^{(2)}a_2^{(1)} + \cdots

Their local derivatives are:

\frac{\partial z_1^{(2)}} {\partial a_1^{(1)}} = w_{11}^{(2)}

and:

\frac{\partial z_2^{(2)}} {\partial a_1^{(1)}} = w_{21}^{(2)}

Using:

\delta_1^{(2)} = \frac{\partial E} {\partial z_1^{(2)}}

and:

\delta_2^{(2)} = \frac{\partial E} {\partial z_2^{(2)}}

the total accumulated gradient becomes:

\boxed{ \frac{\partial E} {\partial a_1^{(1)}} = \delta_1^{(2)}w_{11}^{(2)} + \delta_2^{(2)}w_{21}^{(2)} }

The gradients from all downstream paths are added together.

This allows the network to determine how an early neuron's shared activation affected every connected neuron in the following layer.

The hidden neuron's error signal is then:

\delta_1^{(1)} = \frac{\partial E} {\partial z_1^{(1)}}

Therefore:

\boxed{ \delta_1^{(1)} = \left( \delta_1^{(2)}w_{11}^{(2)} + \delta_2^{(2)}w_{21}^{(2)} \right) \operatorname{ReLU}' \left( z_1^{(1)} \right) }

Updating Bias Parameters

Bias parameters are optimized through backpropagation in the same way as weights.

Consider a bias parameter in layer $3$ :

z_1^{(3)} = \sum_i w_{1i}^{(3)}a_i^{(2)} + b_1^{(3)}

Differentiating the linear combination with respect to the bias gives:

\frac{\partial z_1^{(3)}} {\partial b_1^{(3)}} = 1

Using the chain rule:

\frac{\partial E} {\partial b_1^{(3)}} = \frac{\partial E} {\partial z_1^{(3)}} \cdot \frac{\partial z_1^{(3)}} {\partial b_1^{(3)}}

Since:

\delta_1^{(3)} = \frac{\partial E} {\partial z_1^{(3)}}

we obtain:

\frac{\partial E} {\partial b_1^{(3)}} = \delta_1^{(3)}\cdot1

Therefore:

\boxed{ \frac{\partial E} {\partial b_1^{(3)}} = \delta_1^{(3)} }

Because the local derivative of the pre-activation with respect to its bias is always $1$ , the bias gradient is equal to the neuron's error signal.

Unlike a weight gradient, it is not multiplied by an incoming activation.

Key Takeaways

Backpropagation calculates the gradients required to train a neural network's parameters.
The algorithm uses the chain rule to decompose complex derivatives into manageable local derivatives.
For a Sigmoid output paired with Binary Cross-Entropy, the output error signal simplifies to:
$\delta^{(L)} = \hat{y}-y$
A weight gradient is the destination neuron's error signal multiplied by the activation entering through that weight.
Gradients from multiple downstream paths are added together during the backward pass.
A bias gradient equals the neuron's error signal because its local derivative is $1$ .
Backpropagation computes gradients, while an optimizer such as Gradient Descent uses those gradients to update the parameters.

Paired with Gradient Descent, backpropagation provides the computational mechanism that allows neural networks to learn from their errors.