What Changed After the Model Was Fitted?

A model tested on one mix of conditions may later face another. Comparing the incoming data with the reference data helps us notice changes without pretending that every change proves an accuracy failure.

12 sections · 3 questions · 2 coding challenges

1 / 12

The arrow keys move between sections too.

When Training and Use No Longer Share the Same Distribution

BOTIE

A predictor's evaluation can stop describing its environment when the frequencies or relationships in incoming data change. Detecting that mismatch is a separate problem from fitting the original predictor.

Lipton, Wang, and Smola's 2018 work studies label shift, where class proportions change under specific assumptions, and methods for detecting and correcting it. Our roof-category comparison is a simpler input-distribution audit. It is not their correction algorithm and does not assume that every observed change is label shift.

Sources for this history

Lipton, Wang, and Smola: Detecting and Correcting for Label Shift (2018)

Are We Still Predicting for the Same Greenhouses?

Botie's model was developed when most records came from glass-roofed greenhouses. The site expands, and plastic roofs now dominate the incoming reports. Even if the model code never changes, the population it sees has changed.

We need to notice that change before assuming the old evaluation still describes current performance. First we will compare simple category proportions. Then we will distinguish a changed input mix from a measured decline in prediction quality.

Both illustrative reports use the same roof-category order.

Step 1 of 3

Keep category meanings aligned

Both reports use the order glass, plastic, mesh. Counts cannot be compared until that order agrees.

An invented example for this lesson.
Same category	Reference counts	Current counts
Category 1	80	40
Category 2	20	60
Category 3	0	0

Category 1
Reference counts
80
Current counts
40
Category 2
Reference counts
20
Current counts
60
Category 3
Reference counts
0
Current counts
0

Step 2 of 3

Compare shares, not just volumes

Dividing each report by its own total removes a change caused only by collecting more rows.

An invented example for this lesson.
Same category	Reference shares	Current shares
Category 1	0.8	0.4
Category 2	0.2	0.6
Category 3	0	0

Category 1
Reference shares
0.8
Current shares
0.4
Category 2
Reference shares
0.2
Current shares
0.6
Category 3
Reference shares
0
Current shares
0

Step 3 of 3

Measure how much share moved

The loss of share in glass is balanced by the gain in plastic. Counting both sides without adjustment would count the same movement twice.

Absolute changes: [0.4, 0.4, 0]
Total variation distance = (0.4 + 0.4 + 0) / 2 = 0.4

Half the summed absolute share changes counts the mass that moved, without counting its departure and arrival twice.

Shares let us compare the composition of different-sized reports

Normalisation removes volume differences. The remaining distance measures how the supplied category shares differ.

Did the mix of roofs change, or just the report size?

Double all the counts in one report before changing a single category's share. Comparing proportions should ignore the first change and respond to the second.

BOTIE

Botie needs to compare roof-category shares in a reference report and a new report. Both use the same category order.

Choose an example

Reference category counts

Current category counts

Distance between the category distributions

0.4

Follow the operations

1. Current category counts
1. 040
2. 160
3. 20
2. Add a list
```
100
```
3. Divide
1. 00.4
2. 10.6
3. 20
4. Reference category counts
1. 080
2. 120
3. 20
5. Add a list
```
100
```
6. Divide
1. 00.8
2. 10.2
3. 20
7. Subtract
1. 0-0.4
2. 10.4
3. 20
8. Absolute value
1. 00.4
2. 10.4
3. 20
9. Add a list
```
0.8
```
10. Set a constant
```
2
```
11. Divide
```
0.4
```
12. Result
```
0.4
```

Both reports must have positive totals and aligned, nonnegative counts. This is a distribution audit, not a performance estimate or an automatic retraining rule.

1. Compare Like Categories Over Clear Windows

Choose a reference period and a current period. Keep the category vocabulary and collection rules consistent. A changed logging system can look like a changed population, so check how each report was produced.

Counts depend on how many observations were collected. Proportions describe the mix within each report.

Reference shares = [80, 20, 0] / 100 = [0.8, 0.2, 0]
Current shares = [40, 60, 0] / 100 = [0.4, 0.6, 0]

Both totals must be positive. An empty report has no empirical category distribution to compare.

3. Keep Different Kinds of Change Separate

Input shift concerns changed feature distributions. Label shift concerns changed outcome proportions under an assumption about class-conditional inputs. A changed relationship between inputs and outcomes is often called concept drift.

Our category audit can see roof proportions change. It cannot establish whether the temperature relationship changed, whether accuracy fell, or which assumption about outcomes holds.

A stable input histogram does not prove performance is stable either. Relationships or labels can change without a visible shift in this particular feature.

4. Connect the Alert to an Investigation

When the outcomes arrive, check performance within the affected groups. We can also investigate whether the collection process changed and whether these are the conditions the model was intended to handle.

An alert threshold needs to allow for ordinary sampling variation and seasonal changes. Investigating an alert costs time too. One unusual small batch is a reason to look more closely, rather than an instruction to retrain immediately.

The workshop calculates a transparent category distance. Its labels and proportions stay visible so the number has an interpretation.

5. The Overall Error Can Rise Without Either Group Getting Worse

Suppose a model has always found plastic-roofed greenhouses harder than glass-roofed ones. If plastic roofs become more common, the overall error rises even when the error within each group stays unchanged.

Report the changing mixture beside within-group performance. Otherwise we may attribute the change to a broken model when the immediate cause is a different population. That change can still require action if the new population is the one we must serve.

Change the mixture, keep group errors fixed

The supplied values are illustrative mean absolute errors for glass and plastic roofs.

Python

import numpy as np
error = np.array([1., 4.])
old_share = np.array([.8, .2])
new_share = np.array([.2, .8])
print(round(float(old_share @ error), 2))
print(round(float(new_share @ error), 2))

Output

1.6
3.4

The model now serves more of the group it handles poorly. Group performance and overall performance answer complementary questions.

6. An Input Alarm Does Not Establish a Prediction Failure

A shifted feature distribution is evidence that the incoming data differ from the reference. It does not by itself establish that the target relationship changed, or that performance fell. A unit conversion, changed reporting policy and genuine environmental change need different responses.

Inspect schema and measurement changes first, then obtain outcome labels where possible. Retrain only through a validated procedure. Automatically fitting on a shifted batch can teach the model a temporary fault or contamination instead of a useful new relationship.