Backprop: computation graph direction terminology

Summary: The computation graph is a tree rooted at the output; “children” means inputs, and “upstream/downstream” describe gradient flow direction, which is the opposite of data flow direction.

downstream gradient = local gradient × upstream gradient

Graph direction terminology is genuinely confusing!

The computation graph is a tree rooted at the output.

$$\begin{array}{l}\text{o}\text{ (root / output)}\\ \text{└── tanh}\\ \text{└── n}\text{ (= x1w1 + x2w2 + b)}\\ \text{├── b}\text{ (leaf)}\\ \text{└── x1w1 + x2w2}\\ \text{├── x1w1}\\ \text{│}\text{├── x1}\text{ (leaf)}\\ \text{│}\text{└── w1}\text{ (leaf)}\\ \text{└── x2w2}\\ \text{├── x2}\text{ (leaf)}\\ \text{└── w2}\text{ (leaf)}\end{array}$$

The structure may be more familiar as an image. Here is the forward pass (Left-to-Right):

• Leaf nodes on the left
• Root / output node on the right

---

With that structure in mind, it’s useful to think of backpropagation graph terminology from Right-to-Left:

Takeaways

Term	Node(s) “location”	Means
Root	Rightmost	The output node (o or L) — where backprop starts
Leaf node	Leftmost	A node with no inputs (_prev is empty) — the raw data and weights
Children	Immediately left	A node’s inputs — the nodes in _prev that were used to compute it
Upstream	Left of it	Toward the inputs / leaves — where gradients go to
Downstream	Right of it	Toward the output / root — where gradients come from

downstream gradient = local gradient × upstream gradient

So, confusingly, upstream and downstream mean the opposite of everyday English:

• Forward pass: Data flows “upstream” toward the output
  • leaf → root (i.e. left → right)
• Backward pass: Gradients flow “downstream” toward the inputs
  • root → leaf (i.e. right → left)
• The phrase “backprop deposits the gradient into its children” means:
  • the _backward of n = x1w1 + b takes n.grad and writes into x1w1 and b — the nodes that were used to produce n.

See also

• backpropagation

Sources

• Backpropagation in RNN Explained
• 02_nn_data_structs_and_forward_pass
• 04_backprop_train_a_neuron
• 05_backprop_backward_method