Large Language Models
Give attention a place in the block
BOTIE
Examples, experiments and workspace controls
Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.
Try your own inputs
Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.
Context for this challenge
Normalise the state rows, calculate scaled attention, and add its mixture to the incoming states.
Each token needs access to other positions without passing every relationship through a chain of recurrent states. We have already built attention. Let's place it inside a small block with normalisation, a feed-forward transformation, and residual paths that preserve the incoming information.
Inspect the component calculation
Add(
First contribution: Token states
Second contribution: Matrix multiplication(
Left matrix: Softmax within each row(
Rows of scores: Divide(
Amount: Matrix multiplication(
Left matrix: Normalise each row(
Rows of activations: Token states
)
Right matrix: Swap rows and columns(
Matrix: Normalise each row(
Rows of activations: Token states
)
)
)
Divisor: Square root(
Squared value: 2
)
)
)
Right matrix: Normalise each row(
Rows of activations: Token states
)
)
)Examples and results
- Token states
1 3 4 2 - Required result
0.1116 3.8884 4.8884 1.1116
Try a fresh construction
This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.
Stuck? Inspect a worked construction
The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.
Assumptions and test precision
This two-dimensional teaching block uses identity query, key and value projections and a single feed-forward matrix. Full transformer blocks use learned projections, multiple heads, and a larger feed-forward sublayer. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.
Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.
Shared pattern, different incoming states
This check uses the pieces from “Connect attention and feed-forward residual paths”.
BOTIE
- Token states
2 4 1 3 - Feed-forward matrix
0 0 0 0
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
Which operation mixes positions, and which changes features within each position?
Keep token axes, feature axes, and residual branch inputs distinct when assembling a larger block.
This two-dimensional teaching block uses identity query, key and value projections and a single feed-forward matrix. Full transformer blocks use learned projections, multiple heads, and a larger feed-forward sublayer.
Work through the Python challenges →