Backpropagation

Summary: The algorithm that efficiently computes the gradient, $\nabla C_{\theta }$, of the cost-function with respect to every weight and bias, $\theta ,$ in a neural-network, enabling gradient-descent.

Notes on indexes:

• Superscripts on each neuron are the layer it’s on. Subscript indicates which neuron it is.
• Superscripts on each parameter (weight or bias) indicate the layer they are feeding INTO. Subscripts indicate which neuron they came from and are going to (indexing is backwards, so it’s going-to ← coming-from).

Intuition

For a single training example, consider the output neuron that should be active. Three levers can increase its activation:

1. Increase its bias — constant, predictable shift to the weighted sum.
2. Increase weights connected to high-activation input neurons — because the weight update is proportional to $a^{(L-1)}$, nudging a weight has more effect when the input neuron’s activation is large. This is the Hebbian echo: “neurons that fire together wire together.”
3. Change previous-layer activations — you can’t adjust them directly (they’re determined by earlier weights/biases), but you record the desired changes and propagate them backward to the parameters that do control those activations.

Repeat this layer by layer, from output back to input — that’s backpropagation.

Backpropagation computation tree for a single training example

• Tree diagram showing $w^{(L)}\to z^{(L)}\to a^{(L)}\to C_0$, with arrows labelled by the partial derivatives along each edge (assumes a simple network with only 1 neuron per layer).
• The cost of this one training example is $C_0=(a^{(L)}-y{)}^2$
  • where $a^{(L)}$ is the actual output of the last layer, $L$, of the the network.
  • and $y$ is the desired output from that layer

• As a reminder, this last activation $a^{(L)}$ is determined by a weight, a bias, and the previous neuron’s activation, all pumped through some special nonlinear function like a sigmoid or a ReLU.

$a^{(L)}=\sigma \left(w^{(L)}{a}^{(L-1)}+b^{(L)}\right)$

• For convenience, the weighted sum is called a weighted input, $z$, with the same superscript as the activation:

$\begin{array}{rl}{z}^{(L)}& =w^{(L)}{a}^{(L-1)}+b^{(L)}\\ a^{(L)}& =\sigma \left(z^{(L)}\right)\end{array}$

• The weight, $w^{(L)}$, the previous activation, $a^{(L-1)}$, and the bias, $b^{(L)}$, together let us compute $z^{(L)}$
  • This lets us compute $a^{(L)}$
    • This, along with the constant $y$, lets us compute the cost, $C_0$.

The calculus

Single path (one neuron per layer)

Per example above, define: $z^{(L)}=w^{(L)}{a}^{(L-1)}+b^{(L)}$, $a^{(L)}=\sigma (z^{(L)})$, $C_0=(a^{(L)}-y{)}^2$.

Recall, chain rule (expand)

By the chain rule (See original source for detailed derivation), how sensitive is $C_0$ to small changes in the weight $w^{(L)}$:

$$\frac{\partial C_0}{\partial w^{(L)}}=\underset{\text{input activity}}{\underbrace{a^{(L-1)}}}\cdot \underset{\text{activation slope}}{\underbrace{{\sigma }^{\prime }(z^{(L)})}}\cdot \underset{\text{prediction error}}{\underbrace{2(a^{(L)}-y)}}$$

Factor	Meaning
$a^{(L-1)}$	Weight matters more when input neuron is active
${\sigma }^{\prime }(z^{(L)})$	Sigmoid saturation kills gradients; ReLU avoids this for positive $z$
$2(a^{(L)}-y)$	The further off the prediction, the stronger the signal

For the bias ($C_0$ sensitivity to $b^{(L)}$): same chain but $\partial z/\partial b=1$, so one fewer factor.

Propagating backward

The derivative w.r.t. $a^{(L-1)}$ swaps the first factor for $w^{(L)}$ (See original source). This gives the cost’s sensitivity to the previous activation, which is then used to compute derivatives for the weights/biases in that earlier layer — the recursive step.

Multiple neurons per layer

The single-neuron derivation above generalises cleanly. What changes is the notation — there are more indices to track — and one genuinely new idea at the end.

Notation

Use $j$ to index neurons in layer $L$ and $k$ to index neurons in layer $L-1$. The weighted sum feeding neuron $j$ is now a sum over all $k$ source neurons:

$z_j^{(L)}={\sum }_k{w}_{jk}^{(L)}{a}_k^{(L-1)}+b_j^{(L)}$

The cost for one training example sums over all output neurons:

$C_0={\sum }_j(a_j^{(L)}-y_j{)}^2$

Weight derivative (structurally identical)

The chain rule for a specific weight $w_{jk}^{(L)}$ follows the exact same three-factor pattern as the single-neuron case — because $w_{jk}$ only affects neuron $j$, not any other neuron in the layer:

$$\frac{\partial C_0}{\partial w_{jk}^{(L)}}=\underset{=a_k^{(L-1)}}{\underbrace{\frac{\partial z_j^{(L)}}{\partial w_{jk}^{(L)}}}}\cdot \underset{={\sigma }^{\prime }(z_j^{(L)})}{\underbrace{\frac{\partial a_j^{(L)}}{\partial z_j^{(L)}}}}\cdot \underset{=2(a_j^{(L)}-y_j)}{\underbrace{\frac{\partial C_0}{\partial a_j^{(L)}}}}$$

Same structure, just with subscripts $j,k$ tracking which weight we mean.

Activation derivative (the new idea)

Here is where multi-neuron networks differ. In the single-neuron case, $a^{(L-1)}$ influenced $C_0$ through exactly one path ($z^{(L)}\to a^{(L)}\to C_0$). Now, $a_k^{(L-1)}$ feeds into every neuron $j$ in layer $L$, creating multiple paths to the cost:

$a_k^{(L-1)}⟶z_0^{(L)},z_1^{(L)},\dots ⟶C_0$

To capture the total sensitivity, sum the chain-rule contribution from each path:

$$\frac{\partial C_0}{\partial a_k^{(L-1)}}=\sum _j\underset{=w_{jk}^{(L)}}{\underbrace{\frac{\partial z_j^{(L)}}{\partial a_k^{(L-1)}}}}\cdot \frac{\partial a_j^{(L)}}{\partial z_j^{(L)}}\cdot \frac{\partial C_0}{\partial a_j^{(L)}}$$

This is the only genuinely new equation. Once you have $\frac{\partial C_0}{\partial a_k^{(L-1)}}$ for every neuron $k$ in layer $L-1$, you repeat the same process to get derivatives for the weights and biases in layer $L-1$, and so on backward through the network.

Full cost

Average over all training examples: $\frac{\partial C}{\partial w}=\frac{1}{n}{\sum }_k\frac{\partial C_k}{\partial w}$.

Sources

• src-3b1b-neural-networks-ch4
• src-3b1b-neural-networks-ch5
• Also see: Neural networks and deep learning - Chapter 2: How the backpropagation algorithm works

TODO: Deep dive into backpropagation algorithm

1. Input x: Set the corresponding activation a1 for the input layer.
2. Feedforward: For each l=2,3,…,L compute zl=wlal−1+bl and al=σ(zl).
3. Output error δL: Compute the vector δL=∇aC⊙σ′(zL).
4. Backpropagate the error: For each l=L−1,L−2,…,2 compute δl=((wl+1)Tδl+1)⊙σ′(zl).
5. Output: The gradient of the cost function is given by ∂C∂wljk=al−1kδlj and ∂C∂blj=δlj.

$$\begin{array}{rl}{\delta }^L& ={\nabla }_{a}C\circ {\sigma }^{\prime }(z^L)\\ {\delta }^l& =((w^{l+1}{)}^T{\delta }^{l+1})\circ {\sigma }^{\prime }(z^l)\\ \frac{\partial C}{\partial b_j^l}& ={\delta }_j^l\\ \frac{\partial C}{\partial w_{jk}^l}& =a_k^{l-1}{\delta }_j^l\end{array}\text{\hspace{1em}(BP1)(BP2)(BP3)(BP4)}$$