Take the 6-token sentence “the cat sat on the mat” and look at what one attention layer outputs at the final position. The output vector at mat is not a fixed function of mat alone. It is a weighted sum of vectors from several positions, and the weights come from the tokens themselves. Edit one token, say replace cat with dog, and the weights in some rows shift, so the outputs shift with them.
Two extremes frame the behavior. If a row puts weight 1.0 on a single position, the output is a copy of that position’s vector. If a row spreads weight evenly, say 1/6 on each position, the output is the plain average. Real rows sit between these, and which one occurs depends on the content. Compare the weight rows before and after the edit, and note which rows changed and which stayed fixed. Let’s look at the 6×6 weight matrix for both variants:
The behavior above has a compact definition. Self-attention computes, for each position , an output
where is the value vector derived from token , and is a non-negative weight with for each row. Each row of the matrix is therefore a convex combination of value vectors. A copy is the case where one is near 1, and a blend is the case where several are non-negligible.
What makes this different from a fixed linear mixing layer is that is computed from the data. A fixed mixer would apply the same 6×6 matrix to any sentence. Here the matrix is a function of the input, so changing cat to dog can change the matrix itself, not only the vectors being mixed. We will derive how the weights are produced from queries and keys in the next lessons. For now, only the structure matters.
This is why attention resembles retrieval inside the sequence. Each position issues a content-based lookup against every other position, and the result is a soft read, a weighted mix of what was found. A sharp row behaves like a dictionary lookup that returns one entry. A flat row behaves like a pooled summary.
The weights and values depend only on token content, so the mechanism has a limit. Consider two orders of our tokens.
A: the cat sat on the mat
B: mat the on sat cat the
Suppose the 6 vectors carry no position information. Order B is a permutation of A. Each token’s row of weights is determined by that token’s content against the same set of contents, so each token receives the same output in both orders. The outputs are the same vectors, merely listed in a different order. This property is permutation equivariance. Any pooled readout, such as summing outputs, is fully invariant.
The consequence is concrete. A task that depends on order, such as distinguishing “the cat sat on the mat” from “the mat sat on the cat”, cannot be solved, because both orders hold the same multiset of tokens and produce the same per-token outputs. Positional information must be injected into the token representations, which is the subject of a later lesson. Try the following three checks against the two orders above: