Does This Gradient Describe the Loss We Actually Run?

Botie moves a model weight in the direction the training code recommends, yet the prediction error grows. Before replacing the optimiser, can we check whether its reported slope agrees with the loss we actually evaluate? The loss is the number our chosen rule assigns to the prediction errors.

For one weight, the derivative describes that local slope. With several weights, their derivatives form the gradient. We can check one of those claims by moving the weight a little and running the loss again. We will keep the two sides visible, because an average can conceal a change in behaviour between them.

Botie’s gradient checker reports zero after averaging two incompatible sides of a loss.

{
  "left": 3.9,
  "right": 4.1,
  "central": 4
}

Begin with a tiny model whose prediction is one adjustable weight. Its first loss is squared error relative to a fixed target. We can run that loss at the current weight and at a slightly larger one without needing to trust any derivative code.

Divide the observed loss change by the weight change. This estimates the slope over that finite interval. Holding the target and all other inputs fixed lets us attribute the change to the weight we moved.

A derivative describes the local slope as the interval shrinks at a differentiable point. A finite interval can cross a region where the slope changes, which is why we will inspect both sides.

right slope = (loss(w + h) - loss(w)) / h
left slope = (loss(w) - loss(w - h)) / h

Use a positive step of the same size on the left and right before comparing them.

A central difference compares the right and left perturbed evaluations. It often reduces approximation error for sufficiently smooth functions. On our quadratic, the finite one-sided slopes sit on opposite sides of the derivative and their average gives the exact derivative in exact arithmetic.

That convenient result is a property of this quadratic example. For a general smooth loss, the central approximation still has a finite-step error.

target = 0, w = 2, h = 0.1
left slope = (4 - 3.61) / 0.1 = 3.9
right slope = (4.41 - 4) / 0.1 = 4.1
central difference = (4.41 - 3.61) / 0.2 = 4

Keep the loss definition and perturbation size with the slope so somebody can reproduce the check.

Now change the test loss to the absolute value of weight-minus-target and evaluate exactly at the target. Moving left and moving right approach different slopes. There is no single classical derivative there.

The central difference averages those opposing slopes to zero. Zero is also a subgradient that an optimisation implementation may deliberately choose at this corner. Such a convention is different from proving that a classical derivative exists, and it is not automatically a bug.

absolute loss at w = target = 0, h = 0.1
left slope = -1
right slope = 1
central difference = 0

Avoid judging an automatic derivative solely by a finite-difference check that straddles a nondifferentiable point.

At first, shortening the interval can make a smooth-function approximation more accurate. This is the finite-step approximation error, often called truncation error. Keep shrinking and another problem appears: the two stored loss values become almost equal, so rounding can dominate their subtraction. The moved weight may even round back to the original stored number.

Try several sensible step sizes and inspect where the estimates settle. Their useful range depends on the scale of the weight and on the numerical format used for the calculation.

If neither perturbed evaluation changed, the resulting zero tells us very little about the true slope.

import math
x = 1.0
for h in [.1,.01,.001]:
    estimate = (math.sin(x+h)-math.sin(x-h))/(2*h)
    print(h,round(estimate,6))
print("analytic",round(math.cos(x),6))

0.1 0.539402
0.01 0.540293
0.001 0.540302
analytic 0.540302

Hold the example batch, model parameters other than the perturbed coordinate and relevant model state fixed. Random dropout masks, changing data augmentation or an updated running statistic can change the loss for reasons unrelated to the perturbation.

Use controlled evaluation behaviour for the check and restore any state afterwards. Do not silently redefine the training objective just to make a diagnostic pass. Record what was disabled or fixed.

The workshop uses explicit deterministic scalar losses. A real training system needs a deliberate policy for stochastic and stateful operations.

Perturbing each parameter separately is useful for a small test but expensive for a large model. Another check moves the whole parameter vector along a chosen direction. The resulting scalar slope should agree with the gradient’s dot product with that same direction on a smooth deterministic case.

Do not normalise the direction on only one side of the comparison. Its magnitude determines how far each parameter moves for a given step.

directional difference = (loss(w + h × v) - loss(w - h × v)) / (2 × h)
analytic directional derivative = gradient(w) dot v

A few tested directions cannot certify every coordinate of a large gradient. Keep targeted small fixtures for operations prone to mistakes.

If the estimates disagree on a smooth fixture across sensible step sizes, inspect signs, reductions and parameter indexing. A loss averaged across a batch and a derivative summed across it describe different functions.

If disagreement appears only near an activation boundary or at very small steps, investigate that numerical or mathematical assumption before rewriting a correct derivative. The diagnostic should narrow the cause of a failure.

Finite differences are a development check. They are not a recommendation to replace ordinary gradient-based training with repeated full-model perturbations.