Learning From a Few Records at a Time

Botie’s prediction model has one weight, but the records keep growing. Calculating the effect of every record before each update can become expensive.

Using a smaller batch means we can make an update before reading the entire dataset. But different batches may suggest different directions. We will calculate the contribution from each observation first, then see how averaging a batch affects the update, including a short batch at the end of an epoch.

Mean gradient = (−1 + −4) / 2 = −2.5
Learning rate = 0.1
New weight = 1 − (0.1 × −2.5) = 1.25

Each observation contributes a prediction error and an input value. Follow their contributions into the average gradient, then check how a different batch changes the proposed direction.

1.25

Use a model with one weight and no intercept. Its prediction multiplies each input by the weight. We minimise half the mean squared residual, so one record’s derivative is its residual multiplied by its input.

The half factor cancels the factor from differentiating a square. If you choose ordinary mean squared error instead, its gradient has an additional factor of two. Neither convention is wrong, but the loss and derivative must agree.

Write the objective before diagnosing a surprising update.

For a batch, calculate predictions using the same current weight, then residuals and per-record gradient contributions. Average those contributions before updating once.

x = [1, 2]; y = [2, 4]; weight = 1
Residuals = [-1, -2]
Gradient = mean([-1 × 1, -2 × 2]) = -2.5
New weight = 1 - 0.1 × (-2.5) = 1.25

Updating after each row is a different sequence because later rows see changed weights.

An epoch is a pass through the training records. An update happens after one batch. If the final batch is shorter, average by its actual length, not the usual configured batch size.

Shuffling independent training records can avoid systematic ordering effects. Time-series evaluation must still respect time, and sequence models may need order within sequences preserved. Sampling and evaluation splits solve different problems.

A batch is a two-dimensional feature array in a multi-feature model; targets remain aligned with its rows.

import numpy as np
x, target, weight = 2.0, 6.0, 2.0
rate = 0.1
error = weight * x - target
gradient = error * x
print(error)
print(gradient)
print(weight - rate * gradient)

-2.0
-4.0
2.4

Small batches give cheaper, noisier gradient estimates. Larger batches usually average away more sampling variation, but require more memory per update. Equal epochs do not imply equal numbers of updates.

Compare methods with a declared budget, fixed evaluation records, and repeatable sampling. A lower training loss after one batch does not establish better performance on new data.

The workshop computes a batch update from raw inputs, so the averaging step cannot disappear inside a model call.

Choose one permutation of the training indices, divide it into batches and use each batch to estimate a gradient. The final batch may be shorter. Divide its summed gradient by its actual size, not the usual batch size.

The parameters change between batches, so averaging the gradients observed during an epoch is not generally the full-dataset gradient at one fixed parameter. Keep the distinction between the sampling operation and the sequence of updates.

If memory forces us to accumulate several batches before one update, their means need weights proportional to batch size. Equal weighting gives a short final batch too much influence.

This example holds the parameter fixed and supplies per-record gradients so that only the bookkeeping changes. An accumulated update matches the full mean here because all contributions were evaluated at that same parameter.

import numpy as np
gradient = np.array([1., 2., 3., 10.])
batch_means = np.array([gradient[:3].mean(), gradient[3:].mean()])
sizes = np.array([3,1])
print(float(batch_means.mean()))
print(float(np.average(batch_means, weights=sizes)))
print(float(gradient.mean()))

6.0
4.0
4.0

A smaller batch usually gives a noisier gradient estimate and more updates per epoch. Holding epochs fixed while changing batch size therefore also changes the update count. Comparing runs requires stating which training budget is fixed.

Inspect validation behaviour and optimisation stability. Do not assume that noise is always harmful or always useful. Correlated records within a batch can make it less representative than its row count suggests.