3Blue1Brown — Backpropagation calculus

Summary: Formalises the backpropagation intuition from Ch4 using the chain rule, first on a toy one-neuron-per-layer network, then generalising.

Setup (one neuron per layer)

Define intermediate variables for the last layer:

$z^{(L)}=w^{(L)}{a}^{(L-1)}+b^{(L)},a^{(L)}=\sigma (z^{(L)}),C_0=(a^{(L)}-y{)}^2$

Chain rule decomposition

$$\frac{\partial C_0}{\partial w^{(L)}}=\underset{\partial z/\partial w}{\underbrace{a^{(L-1)}}}\cdot \underset{\partial a/\partial z}{\underbrace{{\sigma }^{\prime }(z^{(L)})}}\cdot \underset{\partial C/\partial a}{\underbrace{2(a^{(L)}-y)}}$$

Each factor has a clear meaning:

• $a^{(L-1)}$: the weight matters more when the input neuron is active (Hebbian echo).
• ${\sigma }^{\prime }(z^{(L)})$: the activation function’s slope gates how much the signal passes through (sigmoid saturation kills gradients here).
• $2(a^{(L)}-y)$: the further off the prediction, the stronger the gradient signal.

Bias derivative

Same chain but $\partial z/\partial b=1$, so it’s simply ${\sigma }^{\prime }(z)\cdot 2(a^{(L)}-y)$.

Propagating to earlier layers

The derivative w.r.t. $a^{(L-1)}$ replaces the first factor with $w^{(L)}$. This gives the sensitivity of the cost to the previous activation, which you then use to repeat the process one layer back — the recursive heart of backprop.

Multi-neuron generalisation

The video argues that “not much changes” — and structurally it doesn’t. The derivation builds up in three steps:

Step 1 — New notation

With multiple neurons per layer, every variable picks up a subscript. Use $j$ for neurons in layer $L$, $k$ for neurons in layer $L-1$. The weighted sum, activation, and cost become:

$z_j^{(L)}={\sum }_k{w}_{jk}^{(L)}{a}_k^{(L-1)}+b_j^{(L)},a_j^{(L)}=\sigma (z_j^{(L)}),C_0={\sum }_j(a_j^{(L)}-y_j{)}^2$

The weight subscript $jk$ means “going to neuron $j$, coming from neuron $k$” — row $j$, column $k$ of the weight matrix.

Step 2 — Weight derivative (same structure)

The chain rule for a specific weight $w_{jk}^{(L)}$ is the same three-factor product as before, just with indices:

$$\frac{\partial C_0}{\partial w_{jk}^{(L)}}=\underset{\partial z_j/\partial w_{jk}}{\underbrace{a_k^{(L-1)}}}\cdot \underset{\partial a_j/\partial z_j}{\underbrace{{\sigma }^{\prime }(z_j^{(L)})}}\cdot \underset{\partial C_0/\partial a_j}{\underbrace{2(a_j^{(L)}-y_j)}}$$

This works unchanged because $w_{jk}$ only affects neuron $j$ — it touches exactly one path through the network.

Step 3 — Activation derivative (the new idea)

In the one-neuron-per-layer case, $a^{(L-1)}$ influenced the cost through a single path. Now, $a_k^{(L-1)}$ feeds into every neuron $j$ in the next layer, so it influences $C_0$ along multiple paths. The total sensitivity is the sum of each path’s chain-rule contribution:

$$\frac{\partial C_0}{\partial a_k^{(L-1)}}=\sum _j\underset{\partial z_j/\partial a_k}{\underbrace{w_{jk}^{(L)}}}\cdot {\sigma }^{\prime }(z_j^{(L)})\cdot 2(a_j^{(L)}-y_j)$$

This is the only genuinely new equation in the multi-neuron case. Once computed, it feeds backward into the next layer’s weight/bias derivatives — the same recursive step as before.

Full cost

Average over all training examples: $\frac{\partial C}{\partial w}=\frac{1}{n}{\sum }_k\frac{\partial C_k}{\partial w}$.

Sources

• Ch. 5 - Backpropagation calculus