Numerical Methods for Learning
A formula can be correct and still give a bad answer on a computer. A calculation may produce a number too large to represent, or a set of measurements may leave several explanations equally possible. We’ll examine those problems before using numerical results to judge a model or a simulation.
Where a fitted number can mislead
A document is represented by a list of numbers, but squaring those numbers can exceed what the computer can represent. Another example gives different parameter choices the same fit. We’ll work out why each result needs closer inspection.
- 01Ready to exploreA Finite Vector Broke Our Normalisation
The encoder returns a larger version of the same direction, and the calculation loses it. Follow the intermediate values to see where that information disappeared.
- 02Ready to exploreA Perfect Fit Left Two Different Answers
Two heater models fit every recorded total. Which experiment would let us find out how much each heater actually contributes?
Checking derivatives and simulated trajectories
A program claims to know how a model’s error will change when a weight moves. We can test that claim by making a small change ourselves. We’ll also compare proposed simulation steps before accepting their results.
- 03Ready to exploreDoes This Gradient Describe the Loss We Actually Run?
A training step is supposed to reduce the loss. Check the reported slope against changes in the running program, including a case where averaging hides the problem.
- 04Ready to exploreWhich Simulation Steps Need More Work?
A cooling curve changes quickly at first and slowly later. Let the proposed calculation show when its time interval needs to be shorter.