Multilayer Perceptron (MLP)

Summary: The simplest neural-network architecture — fully-connected layers stacked in sequence, where information flows strictly from input to output with no cycles.

MLP vs "feed-forward network"

At this level of abstraction, “multilayer perceptron” and “feed-forward neural network” refer to the same thing: layers of neurons where every neuron in layer $l$ connects to every neuron in layer $l+1$, and information flows in one direction only. The name “perceptron” is historical — an MLP has little to do with Rosenblatt’s single-layer perceptron beyond the lineage.

Layer computation

Single neuron

A single neuron $j$ in layer $l+1$ computes two steps. First, the pre-activation — a weighted sum of all inputs from the previous layer, plus a bias:

$z_j^{(l+1)}=w_{j0}^{(l)}{a}_0^{(l)}+w_{j1}^{(l)}{a}_1^{(l)}+\cdots +w_{j,n-1}^{(l)}{a}_{n-1}^{(l)}+b_j^{(l)}$

Then the activation — pass $z_j$ through a nonlinear activation-function $\sigma$ (e.g. ReLU or sigmoid):

$a_j^{(l+1)}=\sigma \left(z_j^{(l+1)}\right)$

Index conventions:

• Superscript $(l)$ — which layer’s parameters are being used. $W^{(l)}$ and $b^{(l)}$ are the weights and biases that transform layer $l$ into layer $l+1$. The superscript lives on the parameters, not the output, because a network has one weight matrix per transition.
• Subscripts on $w_{jk}^{(l)}$ — $j$ indexes the destination neuron (in layer $l+1$), $k$ indexes the source neuron (in layer $l$). Row $j$ of the weight matrix contains all the weights feeding into neuron $j$.
• Subscript on $a_k^{(l)}$ — which neuron within layer $l$.

Matrix form (full layer at once)

The same computation for all neurons in a layer, written as a matrix-vector operation. Separate the two steps:

Step 1 — Pre-activation (linear):

$z^{(l+1)}=W^{(l)}{a}^{(l)}+b^{(l)}$

Step 2 — Activation (nonlinear, element-wise):

$a^{(l+1)}=\sigma \left(z^{(l+1)}\right)$

Or collapsed into one line: $a^{(l+1)}=\sigma (W^{(l)}{a}^{(l)}+b^{(l)})$.

• $a^{(l)}$ — activation vector of layer $l$ (length = number of neurons in layer $l$)
• $W^{(l)}$ — weight matrix for the $l\to l+1$ transition. Rows = neurons in layer $l+1$, columns = neurons in layer $l$. Each row is one neuron’s full set of incoming weights.
• $b^{(l)}$ — bias vector (one entry per neuron in layer $l+1$)
• $\sigma$ — applied element-wise (activation-function: typically ReLU for hidden layers)

Expanded with explicit dimensions

For a layer $l$ with $n$ neurons connecting to a layer $l+1$ with $m$ neurons:

Step 1 — Pre-activation:

$$\underset{[m\times 1]}{\underbrace{{\left[\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_m\end{array}\right]}^{(l+1)}}}=\underset{[m\times n]}{\underbrace{{\left[\begin{array}{cccc}{w}_{10}& w_{11}& \cdots & w_{1,n-1}\\ w_{20}& w_{21}& \cdots & w_{2,n-1}\\ ⋮& ⋮& \ddots & ⋮\\ w_{m0}& w_{m1}& \cdots & w_{m,n-1}\end{array}\right]}^{(l)}}}\underset{[n\times 1]}{\underbrace{{\left[\begin{array}{c}{a}_0\\ a_1\\ ⋮\\ a_{n-1}\end{array}\right]}^{(l)}}}+\underset{[m\times 1]}{\underbrace{{\left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_m\end{array}\right]}^{(l)}}}$$

Inner dimensions cancel ($m\times \mathbf{n}\cdot \mathbf{n}\times 1$), output is $[m\times 1]$ as expected.

Step 2 — Activation: $a_j^{(l+1)}=\sigma (z_j^{(l+1)})$ for each $j=1,\dots ,m$.

The entire network is this two-step operation composed $L$ times — a chain of matrix multiplies, bias additions, and nonlinearities.

Image: Example MLP architecture diagram

A fully connected MLP, with 16 neurons in input layer, two hidden layers with 12 neurons each, and 4 neurons in the output layer

Why layers work

Layers decompose hard problems into sub-problems of increasing abstraction. For image recognition the hope is: pixels → edges → shapes → digits. Each layer performs a relatively simple transformation; the composition is what produces complex behaviour.

Whether the trained network actually learns this neat decomposition is a separate question — in practice, the hidden layers of a simple MLP often learn blobby, hard-to-interpret patterns rather than clean edges and loops (see src-3b1b-neural-networks-ch3).

Key properties

• Expressiveness comes from the nonlinearity. Without an activation-function, stacking layers just produces a single affine map ($W_L\cdots W_{1}x+\text{const}$) — no better than one layer. The nonlinearity is what lets the network represent non-linear decision boundaries.
• Weights are interpretable (in principle). For image inputs, each neuron’s weight vector can be reshaped into the input dimensions and visualised as the pattern the neuron responds to (e.g. positive weights in a strip with negative surround = edge detector).
• Local minima are real. The learned parameters depend on random initialisation and may not correspond to the interpretable decomposition we’d hope for (see src-3b1b-neural-networks-ch3).
• Parameter count grows fast. A fully-connected layer from $m$ to $n$ neurons has $m\times n$ weights + $n$ biases. For the 3Blue1Brown MNIST example (784→16→16→10), that’s 13,002 total parameters. This density is why CNNs use weight-sharing for large spatial inputs.

How it learns

The weights and biases are adjusted by gradient-descent to minimise a cost-function over labelled data. The gradient is computed efficiently via backpropagation.

Limitations

• No spatial awareness. An MLP treats each input dimension independently — it doesn’t know that pixel (3,4) is adjacent to pixel (3,5). A pattern learned in one region of the input doesn’t transfer to another. This is the gap that convolutional neural networks fill.
• No memory. Each input is processed independently. For sequential data (text, audio, time series), recurrent architectures or transformers are needed.
• Scales poorly to high-dimensional inputs. Full connectivity means the parameter count is $O(n\cdot m)$ per layer. For a 224×224 RGB image (150,528 inputs), even one hidden layer of 1,000 neurons would have ~150M parameters — most of them redundant.

MLP blocks inside a transformer

A transformer is interleaved attention blocks and MLP blocks. The MLP block is a two-layer MLP applied independently and in parallel at every token position (no cross-token information flow — that’s attention’s job).

For each token $j$‘s residual-stream vector ${\vec{E}}_j\in {\mathbb{R}}^{d_{\text{embed}}}$:

1. Up-projection (Linear): ${\vec{h}}_j=W_{\uparrow }{\vec{E}}_j+B_{\uparrow }$, where $W_{\uparrow }\in {\mathbb{R}}^{N{d}_{\text{embed}}\times d_{\text{embed}}}$ 1. For GPT-3, $N=4$, so $W_{\uparrow }$ has ~4× as many rows as the embedding dimension. i.e. $W_{\uparrow }\in {\mathbb{R}}^{49,152\times 12,288}$.
2. Nonlinearity: $\vec{h}=\sigma (\vec{h})$ element-wise. The standard choice is ReLU or the smoother GELU (Gaussian Error Linear Unit), which looks like ReLU but with a soft knee near zero.
   1. This ensures cleaner yes/no triggering for when a vector ${\vec{E}}_j$ cleanly “answers” a question (row of $W_{\uparrow }$)
3. Down-projection (Linear): $\Delta \vec{E}=W_{\downarrow }\vec{h}+B_{\downarrow }$, bringing the vector back to the embedding dimension.
4. Residual add: output ${\vec{E}}^{\prime }=\vec{E}+\Delta \vec{E}$.

Rows as questions, columns as answers

Two useful lenses on what the two matrices do:

• Rows of $W_{\uparrow }$ = “questions being asked”.
  • Computing $W_{\uparrow }\vec{E}$ is a pile of dot products between the row vectors of $W_{\uparrow }$ and $\vec{E}$.
  • Each dot product measures how much $\vec{E}$ aligns with some learned direction (analogous to a “question”).
  • With the right bias and ReLU, the $j$-th “neuron” fires if and only if $\vec{E}$ contains some specific combination of features — an AND gate over directions in the embedding space.

$\underset{W_{\uparrow }49,152\times 12,288}{\underbrace{\left[\begin{array}{c}—{\vec{R}}_0^{\top }—\\ —{\vec{R}}_1^{\top }—\\ —{\vec{R}}_2^{\top }—\\ ⋮\\ —{\vec{R}}_n^{\top }—\end{array}\right]}}\underset{\vec{E}12,288\times 1}{\underbrace{\left[\begin{array}{c}|\\ \vec{E}\\ |\end{array}\right]}}+\underset{{\vec{B}}_{\uparrow }49,152\times 1}{\underbrace{\left[\begin{array}{c}{b}_0\\ b_1\\ b_2\\ ⋮\\ b_n\end{array}\right]}}=\underset{\vec{h}49,152\times 1}{\underbrace{\left[\begin{array}{c}{\vec{R}}_0\cdot \vec{E}+b_0\\ {\vec{R}}_1\cdot \vec{E}+b_1\\ {\vec{R}}_2\cdot \vec{E}+b_2\\ ⋮\\ {\vec{R}}_n\cdot \vec{E}+b_n\end{array}\right]}}$

• Columns of $W_{\downarrow }$ = “answers written back”.
  • Computing $W_{\downarrow }\vec{h}$ can be read as summing columns of $W_{\downarrow }$ weighted by the neuron activations (i.e. rescaling then adding columns)
  • Each column is a direction in embedding space that gets added to $\Delta \vec{E}$ whenever its corresponding neuron is active.

$n_0{\vec{C}}_0+n_1{\vec{C}}_1+n_2{\vec{C}}_2+n_3{\vec{C}}_3+n_4{\vec{C}}_4+\cdots$

$=\underset{W_{\downarrow }12,288\times 49,152}{\underbrace{\left[\begin{array}{ccccccc}|& |& |& |& |& & |\\ {\vec{C}}_0& {\vec{C}}_1& {\vec{C}}_2& {\vec{C}}_3& {\vec{C}}_4& \cdots & {\vec{C}}_m\\ |& |& |& |& |& & |\end{array}\right]}}\underset{\vec{h}49,152\times 1}{\underbrace{\left[\begin{array}{c}{n}_0\\ n_1\\ n_2\\ n_3\\ n_4\\ ⋮\\ n_m\end{array}\right]}}+\underset{{\vec{B}}_{\downarrow }12,288\times 1}{\underbrace{\left[\begin{array}{c}|\\ \vec{B}\\ |\end{array}\right]}}$

Together, a single neuron implements: “if the residual stream contains direction X (plus any threshold features encoded in the bias), add direction Y to it”. This is how MLPs are often said to store facts — they’re lookup structures keyed on directions in the embedding space. Worked example: Michael Jordan → basketball. See src-3b1b-llms-ch4-mlps-store-facts for the full walkthrough.

Where most of the parameters live

In GPT-3, each MLP block contributes:

• $W_{\uparrow }$: ~604M params
• $W_{\downarrow }$: ~604M params
• Biases: trivial

~1.2B per block × 96 blocks ≈ 116B params devoted to MLPs — about two-thirds of GPT-3’s total 175B. Attention grabs the spotlight; MLPs hold most of the memory.

Superposition complicates the story

The “rows ask clean questions, columns write clean answers” picture is an idealisation. In practice, individual MLP neurons rarely correspond to single interpretable features — they encode combinations of features in superposition, exploiting the Johnson–Lindenstrauss room in high-dimensional spaces to pack exponentially many near-orthogonal features into the activation vector. Recovering human-interpretable features usually requires tools like sparse autoencoders rather than reading neurons directly.

Sources

• src-3b1b-neural-networks-ch1
• src-3b1b-neural-networks-ch2
• src-3b1b-neural-networks-ch3
• src-3b1b-llms-ch4-mlps-store-facts