Positional Encoding

The same words in a different order can mean something different. A model that receives token vectors needs a way to know where those tokens occur, which positional information provides.

12 sections · 5 questions · 3 coding challenges

1 / 12

The arrow keys move between sections too.

How Would a Model Know Which Word Came First?

BOTIE

The same words can describe different events when we change their order. A collection of word representations alone does not distinguish who acted on whom. If our architecture no longer processes tokens one after another, we need another way to make their positions available.

The 2017 Transformer removed recurrence from its sequence-processing architecture and added positional information to token embeddings. Its paper considered learned positions and used a fixed pattern of sine and cosine values in the reported model.

Think of attaching an address to each token's representation. The content says which token is present; the positional information helps identify where it appears. Different position schemes provide different ways to compare addresses, including relative separations. Position is additional information for the model to use, not a guarantee that it will interpret every ordering correctly.

Sources for this history

Vaswani and colleagues (2017), Attention Is All You Need

The Same Words Can Say Something Different

The dog chased the cat and the cat chased the dog contain the same words but describe different events. A representation listing only which words occurred has lost information.

Position information gives a model access to where each token appears, alongside what token it is.

Step 1 of 4

Hold the words fixed and change their order

The vocabulary entries are unchanged, but the actor and recipient change. Order is information the task may need.

First sentence

Position 1dog
Position 2chased
Position 3cat

Second sentence

Position 1cat
Position 2chased
Position 3dog

The same words in different positions.

Step 2 of 4

A lookup alone does not identify the position

An embedding table returns the same starting row for dog wherever it appears. Something else must supply position if later operations are to distinguish its placement.

Dog at the beginning: Dog embedding
Dog at the end: Dog embedding

Token identity and occurrence position are separate.

Step 3 of 4

Supply a position-dependent description

Associate each sequence position with a vector and combine it with the token vector. The resulting input carries information from both sources. Sinusoidal encoding uses a fixed pattern rather than fitting a separate row here.

Position one: Token vector
Position-one vector
Position two: Token vector
Position-two vector

The layer combines content and position information.

Step 4 of 4

Do not confuse order with permission to look

Position encoding describes where tokens occur. A causal mask determines which tokens may interact. Supplying positions does not, by itself, stop a token from reading future information.

Position encoding: Where is this token?
Causal mask: Which positions may it read?

Two mechanisms answer different questions.

Content and location become available together

The numerical example constructs the position vectors and shows their addition to token embeddings.

A model must still learn to use this information. Encoding position does not prescribe a grammatical interpretation.

Part 1. Keep Content and Position Available

1. Add information without adding coordinates

An ordinary unmasked self-attention operation, without positional information, treats a rearranged sequence as the same vectors in a new order: its outputs are rearranged accordingly. But word order can change a sentence's meaning. A token embedding alone cannot distinguish the first occurrence of a word from its later occurrence. We need to give the model information about position. A causal mask adds ordering constraints too, but does not replace the positional representation used in this example.

What is missing is a vector that depends on where a token sits and on nothing else. We make that position vector the same width as the token vector, then add corresponding entries. This keeps the shape expected by the next layer. It also means the next layer sees a mixture of content and position, and must learn to use that mixture.

Adding a position vector is a design choice. A model could instead learn a table of position vectors, or encode relative positions in another way. We start with a fixed pattern because every value can be reproduced without training another table, and because a position the training data never reached still gets a vector, since the vector is a function of the position rather than an entry looked up in a table.

The same token at three positions

For a four-coordinate vector, the first sine-cosine pair changes faster than the second. At position zero, all the sine entries are zero and the cosine entries are one.

Token vector = [1, 0, 1, 0]
Position 0   = [0, 1, 0, 1]
Combined     = [1 + 0, 0 + 1, 1 + 0, 0 + 1]
             = [1, 1, 1, 1]

Position 1 ≈ [0.8415, 0.5403, 0.0100, 0.99995]
Combined   ≈ [1.8415, 0.5403, 1.0100, 0.99995]

Position 2 ≈ [0.9093, −0.4161, 0.0200, 0.9998]
Combined   ≈ [1.9093, −0.4161, 1.0200, 0.9998]

The repeated token still has the same content vector, but its combined rows differ because of their positions. For these two occurrences of the same token, subtracting the rows cancels the content and leaves the positional difference. With different tokens, the content difference remains too. The next layer must learn to use the mixture; we have not handed it a separate, automatically decoded distance.

Part 2. Building a Position Vector

2. Use pairs that change at different rates

Let p be the position, starting at zero. Let d be the vector width and k identify a pair of coordinates. B controls the spread of frequencies. Each pair has its own rate, one over B raised to a power that grows with k, so the first pair turns by one radian per position and every later pair turns more slowly. With the default base used here, a width of four has two pairs.

B = 10000
angle(p, k) = p / B^(2k/d)
PE[p, 2k]   = sin(angle(p, k))
PE[p, 2k+1] = cos(angle(p, k))

For d = 4 at p = 1:
First pair angle  = 1 / 10000^0   = 1
Second pair angle = 1 / 10000^0.5 = 0.01

A single sine value repeats and can be ambiguous. Pairing sine and cosine gives the phase of one oscillation, which is a point on a circle, and moving forward by a fixed number of positions turns that point by a fixed angle wherever the move started. A layer above can therefore read a distance as a rotation, and the rotation for a given distance is the same everywhere in the sequence.

Using several pairs at different frequencies provides patterns over different distances. The first pair completes a turn every 2π positions, which is about 6.28, and the second needs a hundred times as many. The fast pair on its own would nearly confuse position six with position zero, since by then it has come most of the way round.

Position 0, first pair  = (0, 1)
Position 6, first pair  ≈ (−0.2794, 0.9602)

Position 0, second pair = (0, 1)
Position 6, second pair ≈ (0.0600, 0.9982)

The slow pair has barely moved, and that small movement is what keeps the two positions apart. Lowering the base speeds the slow pairs up. At a base of 2 the second pair’s angle at position one is 1 over the square root of 2, about 0.7071, and its entries become 0.6496 and 0.7602 instead of 0.0100 and 0.99995, which is what the frequency base control in the playground shows. None of this makes the finite-width encoding a guarantee of uniquely identifying arbitrarily distant positions.

3. Add the pattern to every sequence

The SDK constructs the pattern once, freezes it, and broadcasts it over the batch. Position zero has the same positional vector in every sequence, because position means where a token sits inside its own sequence and not which sequence it belongs to. If the vector width is odd, the last coordinate contains a sine value without its paired cosine.

output[batch, position, coordinate]
    = input[batch, position, coordinate] + pattern[position, coordinate]

The pattern has no trainable parameters. Adding a constant has derivative one with respect to the input, so the backward pass returns the arriving input gradient unchanged, and a training step leaves the layer exactly as it was.

Part 3. Order Is Different from Permission to Look

A positional encoding gives information about location. A causal mask restricts which locations can contribute. They solve different problems and can be used together. The attention playground lets you enable them separately, and the two controls change different things, since the mask changes which columns of a row may be used and the encoding changes what every row holds.

Two identical tokens, with and without positions

Take the attention lesson’s default example, the tokens red, blue, red with projections that leave every vector unchanged, and read the row of shares each red token produces over the three tokens.

Positions off, mask off
  first red = [0.3837, 0.2327, 0.3837]
  last red  = [0.3837, 0.2327, 0.3837]

Positions on, mask off
  first red = [0.3349, 0.4053, 0.2599]
  last red  = [0.2013, 0.1547, 0.6440]

Positions off, mask on
  first red = [1, 0, 0]
  last red  = [0.3837, 0.2327, 0.3837]

Without positions the two red tokens are the same input, so their rows are identical and nothing downstream can tell them apart. Adding the pattern makes them different inputs, and the rows differ with no training at all. The mask does something else. It leaves the last red’s row exactly as it was, because that token may read everything before it, and it empties the entries the first red is not allowed to use while the remaining shares still total one.

Being able to calculate a position vector for a longer sequence does not guarantee that a trained model will work well at that length. Position information also does not impose grammatical understanding. It gives the model information from which useful order-dependent behavior can be learned.

The sinusoidal construction appears in Attention Is All You Need. The parameter called wavelength in this SDK supplies the frequency base B in the equations above.

The Same Words Can Describe Different Events

Compare the dog chased the cat with the cat chased the dog. A collection of word vectors contains the same words, but their positions change who did what. A model needs a signal that distinguishes those arrangements.

Without an order signal or order-dependent constraint, self-attention processes a reordering as a corresponding reordering of its inputs. Content similarity alone does not assign first, second or later roles. Causal masking supplies an ordering constraint, while positional information can describe locations and distances more directly.

Add a Position Signal without Creating More Token Rows

An additive position vector has the same width as a token embedding. The model receives their sum at each sequence position. This preserves the table shape while making two occurrences of the same word distinguishable by location.

The addition does not attach a human-readable position label to a dedicated coordinate. Later learned operations must use the combined representation.

One word at two positions

The position vectors below are supplied illustrative vectors, not a fitted positional table.

Python

import numpy as np
word = np.array([2.,1.])
position = np.array([[0.,1.],[1.,0.]])
encoded = word + position
print(encoded.tolist())
print(encoded.shape)

Output

[[2.0, 2.0], [3.0, 1.0]]
(2, 2)

Both rows began with the same word vector. Their differing position contributions make their resulting representations different.

A Position Rule and a Trained Context Length Are Different Limits

A sinusoidal formula can be evaluated at positions beyond those seen during training. That does not establish that the trained model will behave well there. Learned absolute tables can also have an explicit maximum index.

Relative-position methods and rotary methods change how positional relationships enter attention. They do not remove memory cost, training-distribution limits or the need to evaluate long-context behaviour. Distinguish a mathematically defined position from a context length the full model has learned to use.