What Are the Errors Trying to Tell Us?

The temperature errors average to zero, but the greenhouse is still too hot some mornings. Keep each mistake beside the reading that produced it and work out what the average concealed.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Checking What a Fitted Equation Left Behind

BOTIE

Fitting a regression equation leaves a difference at each observation. Statistical model checking examines those differences because a fitted average can conceal a systematic pattern. The NIST engineering handbook documents checks involving residual plots and model assumptions.

Our greenhouse readings are invented. We use the same diagnostic idea: keep the part the prediction failed to explain, then ask whether it changes with temperature, time or another recorded condition.

Sources for this history

NIST: Are the model residuals well-behaved?

How Can Zero Average Error Still Be a Problem?

Botie predicts the greenhouse temperature before the heaters switch on. On one morning the prediction is too low. On another it is too high by the same amount. If we add those mistakes, they cancel. The plants still experienced both mornings.

We need to keep two questions separate. Does the model usually miss in one direction? How large are its misses? Once we can answer those, we can look for conditions that make the model struggle.

Invented temperatures in degrees Celsius. These predictions are supplied for diagnosis; the workshop does not train a model.

Step 1 of 2

Two mistakes cancel on paper

Subtract each prediction from its observed temperature. Keep the morning attached to both values.

Positive means the prediction was too low. Negative means it was too high.
Morning	Observed	Predicted	Residual
A	12	10	+2
B	16	18	-2

A
Observed
12
Predicted
10
Residual
+2
B
Observed
16
Predicted
18
Residual
-2

Positive means the prediction was too low. Negative means it was too high.

Step 2 of 2

A large miss changes the comparison

One new morning has a much larger error. Compare quantities in the original temperature units.

The same MAE can accompany a very different distribution of errors.
Residual set	Mean residual	MAE	RMSE
[2, -2]	0	2	2
[0, 0, 6]	2	2	3.4641

[2, -2]
Mean residual
0
MAE
2
RMSE
2
[0, 0, 6]
Mean residual
2
MAE
2
RMSE
3.4641

The same MAE can accompany a very different distribution of errors.

A score is a question about errors

The intermediate residuals explain the report. A summary alone cannot locate a broken condition.

Keep direction before averaging

Use the observed temperatures and predictions to inspect the errors your construction produces.

BOTIE

Zero signed error has concealed mistakes in both directions.

Choose an example

Observed temperatures

Predicted temperatures

Residual summary

{
  "bias": 0,
  "mae": 2,
  "rmse": 2
}

Follow the operations

1. Observed temperatures
1. 012
2. 116
2. Predicted temperatures
1. 010
2. 118
3. Observed minus predicted
1. 02
2. 1-2
4. Mean
```
0
```
5. Keep the size of each error
1. 02
2. 12
6. Mean
```
2
```
7. Square
1. 04
2. 14
8. Mean
```
4
```
9. Square root
```
2
```

10. Label the three error measures

{
  "bias": 0,
  "mae": 2,
  "rmse": 2
}

11. Result

{
  "bias": 0,
  "mae": 2,
  "rmse": 2
}

Supplied predictions make the arithmetic inspectable. These tiny fixtures do not estimate deployment performance.

1. Give every miss a direction

A residual is the observed value minus the prediction. We will use that convention throughout this lesson. A positive residual means Botie predicted too little heat; a negative one means it predicted too much. Some software reports prediction minus observation, so check the convention before interpreting a plot.

Store the observation ID with its residual. Rows shuffled in one array but not the other create meaningless differences even if their shapes match.

Residuals = observed - predicted
[12, 16] - [10, 18] = [2, -2]
Mean residual = (2 - 2) / 2 = 0

Here bias means mean signed error on this sample, not the statistical bias of an estimator across repeated training samples. Zero does not say the predictions were close.

2. Measure size without cancellation

Take the absolute value of each residual to remove its sign. Their mean is mean absolute error, or MAE. It has the same units as the observations.

Root mean squared error, or RMSE, first squares each residual, averages the squares, then takes a square root. Squaring gives large misses more influence. Which measure is useful depends on what an error costs in the actual task.

Residuals: [0, 0, 6]
MAE = (0 + 0 + 6) / 3 = 2
MSE = (0 + 0 + 36) / 3 = 12
RMSE = sqrt(12) ≈ 3.4641

RMSE is not inherently more honest than MAE. Report the measure that matches the decision, and inspect the tail when rare large misses matter.

3. Put the residuals back beside the conditions

Draw residuals against predicted temperature on a shared vertical scale. A horizontal band around zero gives a different clue from a curve or a widening spread. A curve suggests the fitted mean misses a relationship. A widening spread suggests error size changes with conditions. Neither picture identifies the cause by itself.

Now look against time and heater setting. Consecutive misses may come from a lagged effect, a changed sensor or a missing input. Inspect the records before deciding that a more flexible model is the repair.

Useful columns:
observation_id | time | heater_setting | prediction | observed | residual

Use development or out-of-fold predictions for this diagnosis. Once a final test set guides a repair, it has become development evidence.

4. Different questions need different assumptions

Squared-error regression does not require normally distributed outcomes just to calculate predictions. Normality and constant-variance assumptions enter particular inference procedures and interval formulas. Do not turn a diagnostic convention into a claim that every useful model must have Gaussian residuals.

For predictive evaluation, care first about the population and conditions the test represents. Training residuals are often smaller because those observations helped choose the coefficients.

A pattern deserves an investigation. An apparently patternless plot does not prove that every relevant condition has been sampled.

5. Calculate from one-dimensional arrays

Each position below represents one morning. Subtraction acts element by element. Taking an average too early would discard the differences we need to inspect. NumPy reductions return scalars; use those only after creating the residual array.

Keep full precision while calculating. Round for display, and write down the units.

Inspect before reducing

This example prints the individual residuals and their absolute sizes. The challenge will assemble several summaries into a reusable function.

Python

import numpy as np
y = np.array([9., 15., 21.])
pred = np.array([10., 12., 20.])
r = y - pred
print(r.tolist())
print(np.abs(r).tolist())

Output

[-1.0, 3.0, 1.0]
[1.0, 3.0, 1.0]

The first morning was overpredicted. Taking absolute values keeps its size and deliberately loses that direction.

6. Choose the next experiment from the pattern

Suppose the high-temperature mornings are consistently underpredicted. A useful next experiment might add a justified heater interaction or collect more hot-day readings. Record that hypothesis before trying several unrelated changes.

Compare the candidate on the same development cases, including the mornings the baseline handled well. Later we will separate a genuine repair from a favourable sample and from a revision that helps only the largest group.

For multiple outputs, report units and aggregation separately. Averaging temperature errors with electricity-cost errors would need an explicit conversion or task cost.