Word Embedding

Summary: The operation that turns a discrete token ID into a high-dimensional real-valued vector, where directions in the space tend to encode semantic features that the model has found useful during training.

The embedding matrix

Every transformer begins with an embedding matrix $W_E$, with one column per vocabulary token, and one row per embedding dimension:

$W_E\in {\mathbb{R}}^{d_{\text{embed}}\times |V|}$

Embedding a token is literally a column lookup: token ID $i$ becomes column $i$ of $W_E$. No computation, just indexing. The values are learned via backpropagation like any other parameter.

For GPT-3: $d_{\text{embed}}=12,288$ rows and $|V|=50,257$ columns, so $W_E$ holds ~617M parameters — already more than most pre-transformer models had in total.

Abbreviated example of $W_E$ as described in GPT-3

$$\stackrel{\stackrel{\text{All words, }\sim 50\text{k}}{\overbrace{}}}{}$$ $$\begin{array}{ccccccccccc}\text{aah}& \text{aardvark}& \text{aardwolf}& \text{aargh}& \text{ab}& \cdots & \text{zygotic}& \text{zyme}& \text{zymogen}& \text{zymosis}& \text{zzz}\end{array}$$ $${\left[\begin{array}{rrrrrrrrrrr}+1.0& +4.3& +2.0& +0.9& -1.5& \cdots & -9.5& +6.6& +5.5& +7.3& +9.5\\ +5.9& -0.8& +5.6& -7.6& +2.8& \cdots & +2.3& +8.8& +3.6& -2.8& -1.2\\ +3.9& -8.7& +3.3& +3.4& -5.7& \cdots & -0.7& -5.1& -6.8& -7.7& +3.1\\ -7.2& -6.0& -2.6& +6.4& -8.0& \cdots & -7.5& -3.6& -1.7& -8.6& +3.8\\ +1.3& -4.6& +0.5& -8.0& +1.5& \cdots & +3.5& -4.6& +4.7& +9.2& -5.0\\ ⋮& ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮& ⋮& ⋮\\ -3.7& -2.0& -5.7& -6.2& +8.8& \cdots & -8.6& +3.6& -0.9& +0.7& +7.9\end{array}\right]}_{12,288\times 50,257}$$

Directions encode meaning

This is the empirically remarkable fact about learned embeddings: after training, directions in the embedding space correspond to semantic features.

The classic examples (from word2vec-era models but also present in transformer embeddings):

• $\text{king}-\text{man}+\text{woman}\approx \text{queen}$ — a “gender” direction exists.
• $\text{cats}-\text{cat}$ is approximately a “plurality” direction; its dot product with one, two, three, four increases monotonically.
• $\text{Italy}-\text{Germany}+\text{Hitler}\approx \text{Mussolini}$ — the model has learned “country of origin” as a direction it can translate along.

These are not designed. They emerge because representing related concepts along a shared axis lets downstream layers generalise cheaply — the model learns to exploit linearity because linearity is what the rest of the network is good at.

The classic examples are approximate

The “queen” example is famous but imperfect — in real models the nearest neighbour to king − man + woman is often still king. Family relations (father/mother, uncle/aunt) illustrate the idea more cleanly. The underlying claim is directional, not exact-point-arithmetic.

Dot product as alignment

The dot product is the workhorse similarity measure throughout transformers:

$\vec{a}\cdot \vec{b}=\Vert \vec{a}\Vert \Vert \vec{b}\Vert \cos \theta$

Element-wise, it looks like:

$$\vec{a}\cdot \vec{b}=\left[\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]\cdot \left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_n\end{array}\right]=a_1{b}_1+a_2{b}_2+\cdots +a_n{b}_n=\text{constant}$$

• Positive → vectors point in a similar direction (aligned).
• Zero → perpendicular (unrelated).
• Negative → opposing directions.

Every time the transformer asks “how much does X relate to Y?”, the answer is computed as a dot product somewhere:

• Attention uses $Q\cdot K$ to score query–key alignment.
• The MLP up-projection computes $\text{row}\cdot \vec{E}$ to ask “how much does this vector align with a particular learned direction?”
• The unembedding step uses $W_U\vec{E}$, i.e. dot products against one row per vocabulary token, to produce logits.

Beyond words

Inside a transformer, an embedding is not static. As it flows through attention and MLP blocks, the residual-stream at a given position gets progressively refined:

• The vector that started as “king” might end up pointing in a direction that encodes “a king, in Scotland, who murdered the previous king, described in Shakespearean language”.
• The vector for “mole” starts identical in “American shrew mole”, “one mole of CO₂”, and “biopsy the mole” — only context-blending in later layers separates the three meanings.

In this sense the first-layer embedding is only a seed. The interesting representations live later in the stack.

Positional information

create-page Embeddings also have to carry where a token is, not just what it is. The 3b1b series treats positional encoding as “part of the embedding” without detailing the specific scheme — different transformer variants use learned position embeddings, sinusoidal encodings, rotary (RoPE), or ALiBi. Flag for a later source that covers them.

See also

• transformer-architecture — where embeddings sit in the pipeline
• unembedding — the inverse operation at the end
• attention-mechanism — how embeddings get refined
• superposition — why “one direction per concept” isn’t quite the full story

Sources

• src-3b1b-llms-ch2-transformers
• src-3b1b-llms-ch3-attention