Predicting Between Measurements With Uncertainty

Botie has a thermometer at one end of the greenhouse. It reads warmer than usual. Should we expect the next bed to be warmer too? What about the far end, where a door is open?

One reading leaves many possible temperature patterns. We need a way to describe which patterns we consider plausible, then change that description when readings arrive. Nearby positions can share information without pretending they have exactly the same temperature.

Start with the positions and readings in the experiment. Add a thermometer, change how quickly temperatures may vary, and watch both the predicted temperature and the uncertainty change. After that, we will build the table that makes those changes possible.

Training function covariance: 1.0
Measurement noise variance: 0.25
Combined denominator = 1.0 + 0.25 = 1.25

Observed residual: 2
Query-to-training covariance: 0.5
Mean contribution = 0.5 × (2 / 1.25) = 0.8

Posterior mean: 0.8
Latent variance = 1 − ((0.5 × 0.5) / 1.25) = 0.8

Compare the unobserved greenhouse with the prediction after one and then two readings. Change the length scale or noise variance while keeping the readings fixed. The graph shows the temperature mean and a pointwise uncertainty band; the construction later isolates the matrix calculation.

Imagine drawing several temperature curves through the greenhouse before looking at a reading. Some rise towards the sunny wall; some stay close to the usual temperature; others fall near the door. We are describing uncertainty about a whole pattern, not choosing one curve and declaring it correct.

A mean function gives the temperature around which those possibilities are centred. Here we subtract a usual temperature of twenty degrees and work with residuals. A residual of two means two degrees above that baseline. We add the baseline back when presenting a temperature.

Usual temperature: 20 °C
Observed temperature: 22 °C
Residual: 22 - 20 = 2 °C

Using residuals keeps the baseline distinct from what the readings teach us.

If temperatures normally change slowly along the greenhouse, a warm reading should favour warm curves at nearby beds too. If a heater or doorway can cause a sharp local change, that influence should fade more quickly. This assumption about how locations vary together is what the covariance function, or kernel, describes.

The length scale is measured in the same distance units as position. Increasing it connects more distant locations strongly. It does not move a thermometer or supply another observation. In the experiment, first compare a nearby query with a distant one while keeping the reading fixed.

The assumption is chosen before conditioning on the readings. A confident prediction can still be wrong when the assumed pattern is unsuitable.

Use an RBF kernel with prior variance one. Compare every training position with every other training position. Each table entry says how strongly those two unknown residual temperatures vary together under this model.

The diagonal compares a position with itself and therefore contains its prior variance. Off-diagonal entries become smaller as positions move apart. The table contains relationships between function values; it does not contain the measured temperatures.

k(a,b) = exp(-(a-b)² / (2 × length_scale²))
Positions = [0, 2] metres; length_scale = 1 metre
k(0,0) = 1
k(0,2) = exp(-2) ≈ 0.1353
K ≈ [[1, 0.1353],
     [0.1353, 1]]

For n readings, K has shape (n,n). A query-to-training row has n entries, in the same order as the observed residual vector.

A mean and covariance together describe the joint Gaussian distribution of function values at the positions we have chosen. We can ask about another set of positions using the same kernel. This consistency is what lets us describe a Gaussian process rather than one fixed table.

A valid covariance matrix is symmetric: the relationship between positions a and b cannot change when we reverse their names. It must also be positive semidefinite, meaning no weighted combination of its function values has a negative variance. Using a known valid kernel gives us this property; assigning arbitrary similarities does not.

The examples use a zero mean for small residuals, not a claim that greenhouse temperatures are near zero.

Independent measurement noise adds variance to the diagonal of the training covariance. It does not add the same value to every entry. Solving this noisy covariance system tells us how observations contribute after accounting for their redundancy.

A = K + noise_variance × I
A = [[1.25]]
Solve A × alpha = [2]
alpha = [1.6]

Use a linear solve rather than explicitly calculating a matrix inverse.

The query-to-training covariance vector weights the solved observations to produce the posterior mean. To obtain latent variance, subtract the covariance explained by training from the query’s prior variance.

Prediction for a new noisy sensor reading adds its measurement noise to that latent variance. Keep these two types of uncertainty separate.

Mean = k @ solve(A, y) = 0.5 × 1.6 = 0.8
Latent variance = k(query,query) - k @ solve(A,k)
Latent variance = 1 - 0.5 × 0.4 = 0.8

A query with no covariance to training retains its prior mean and variance.

import numpy as np
noisy_covariance = np.array([[2.5]])
observed = np.array([5.0])
query_covariance = np.array([1.0])
coefficients = np.linalg.solve(noisy_covariance, observed)
mean = query_covariance @ coefficients
variance = 2.0 - query_covariance @ np.linalg.solve(noisy_covariance, query_covariance)
print(coefficients.tolist())
print(f'{mean:.1f}')
print(f'{variance:.1f}')

[2.0]
2.0
1.6

A long length scale expects slowly changing temperatures; a short one allows faster local changes. Fit such settings on training data and check predictions on held-out locations. A narrow interval is not trustworthy if the kernel excludes important behaviour.

Dense exact inference requires storing a covariance matrix and solving a system that grows with the training set. Large datasets need approximations or a different model. Numerical jitter can stabilise a solve, but does not replace justified measurement noise.

The construction isolates posterior mean. The second coding task checks uncertainty at an unrelated location.

Place two thermometers close together and give them conflicting readings. A model with almost no observation noise tries to follow both. Allowing more noise tells the model that a reading need not equal the underlying temperature exactly.

The shaded band in the experiment describes uncertainty about that underlying function, using the model’s Gaussian marginal at each position. A future noisy thermometer reading has additional variance. Neither band is a guarantee that every point on the whole curve lies inside it.

Approximate pointwise 95% latent interval:
mean ± 1.96 × sqrt(latent variance)

Observation variance = latent variance + new reading noise variance

Compare the interval at measured positions and between them. Then move beyond the measurements: the mean returns towards the baseline and uncertainty approaches the prior under this kernel.

Suppose the greenhouse contains a local heater. A long length scale may smooth away its effect and still produce a narrow band near existing thermometers. Add a held-out reading at the heater and compare it with the prediction made before revealing it.

Repeat this at several held-out positions. Inspect errors and interval coverage together. Choosing the length scale from these results uses those positions for validation; reserve another set if you want a final assessment. A single covered observation cannot establish calibrated uncertainty.

For a larger dataset, use a stable factorisation such as Cholesky when the noisy covariance is positive definite. Keep a numerical stabiliser separate from the measurement-noise assumption. The small NumPy challenge uses solve so the matrix calculation remains visible.

The aim is to explain where the prediction came from and what could make it unreliable, not simply to draw a smooth curve.