Optimisation and Training
Carry an update direction forward
A gradient with the same sign as memory
Botie's optimiser retains a direction from earlier steps. A current gradient can reinforce that memory or cancel it before any weight change is applied.
BOTIE
The inputs we are comparing
Old velocity
Cancellation
2
Persistent direction
1
Current gradient
Cancellation
-1
Persistent direction
1
Retention
Same in both cases
0.5
Learning rate
Cancellation
0.1
Persistent direction
0.2
Weight
Cancellation
4
Persistent direction
2
Result for cancellation
4
The gradient changes direction before the velocity does
This check uses the pieces from “Combine the stored velocity with today's gradient”.
BOTIE
- Old velocity
4
- Current gradient
-1
- Retention
0.75
- Learning rate
0.2
- Weight
3
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
What happens to the influence of stored velocity when beta is zero?
An optimiser's state and its current gradient are separate inputs to an update.
This uses v_new = beta*v + gradient, with beta in [0,1).
Work through the Python challenges →