← All workshopsRead Next-Token Prediction

Large Language Models

Do not let tomorrow give away the answer

Give future positions the largest scores

Botie's next-token predictor must not consult the future token that supplies its training answer.

BOTIE

For query position 1, what probability vector remains when positions 2 and 3 carry scores 8 and 9 but are unavailable?
The inputs we are comparing

Key positions

Same in both cases

  1. 00
  2. 11
  3. 22
  4. 33

Current query position

Example 1

2

Example 2

1

Attention scores

Example 1

  1. 01
  2. 12
  3. 23
  4. 39

Example 2

  1. 02
  2. 11
  3. 28
  4. 39
Result for example 1
  1. 00.09
  2. 10.2447
  3. 20.6652
  4. 30

For a list use brackets, such as [2, 4]. For a matrix put each row in brackets. Text answers can stay as text.

The first position cannot read ahead

This check uses the pieces from “Make future positions invisible before normalising”.

BOTIE

Botie is processing position zero. Later positions have very large compatibility scores, but their tokens have not become available to this query. Apply the causal mask and return the four attention probabilities in position order.
Key positions
  1. 00
  2. 11
  3. 22
  4. 33
Current query position
0
Attention scores
  1. 0-2
  2. 15
  3. 220
  4. 3100

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why is zeroing future probabilities after an unrestricted softmax insufficient without renormalising?

A causal boundary changes the set over which attention is normalised.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseAutoregressive Generation →
← Transformer Blocks