Do We Need a Learned Model Yet?

A fixed rule already catches some faults. Work out what a learned model would have to improve before paying for its extra work.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Learning Needs Something to Improve On

BOTIE

Google’s Rules of Machine Learning recommends starting without machine learning when the necessary data are not available, and establishing metrics early. A working heuristic can make a service useful while its team learns what the problem actually demands.

For our inspection service, the temperature rule provides that first comparison. We will assign invented costs to missed faults and inspections. Those values explain the calculation; a real maintenance team would need to justify its own costs and any limits that cannot be traded away.

Sources for this history

Martin Zinkevich: Rules of Machine Learning

The Temperature Rule Already Catches Some Problems

Botie can send an inspection request whenever a pump gets unusually hot. It is a simple rule, and it will miss some faults. Before replacing it with a trained model, we should find out where it helps and where it fails.

A model might catch more faults while creating extra inspections. We need a comparison that accounts for both effects, using the same pumps and the same opportunity to act.

This cost model is an invented comparison, not a measured financial return.

Step 1 of 2

Keep the pumps fixed

A fair comparison gives both policies the same cases. Fault labels are known here because this is a retrospective exercise.

The model catches an extra fault but still sends one healthy pump for inspection.
Pump	Fault	Heat rule inspects	Model inspects
A	1	1	1
B	0	1	0
C	1	0	1
D	0	0	1

A
Fault
1
Heat rule inspects
1
Model inspects
1
B
Fault
0
Heat rule inspects
1
Model inspects
0
C
Fault
1
Heat rule inspects
0
Model inspects
1
D
Fault
0
Heat rule inspects
0
Model inspects
1

The model catches an extra fault but still sends one healthy pump for inspection.

Step 2 of 2

Name what each outcome costs

An inspection costs time whether or not a fault is found. A missed fault adds a separate loss.

We assume an inspection catches the fault in this teaching example.
Event	Cost units	When counted
Inspection	3	Decision is 1
Missed fault	20	Fault is 1 and decision is 0
Model upkeep	Separate budget	Only when comparing total ownership

Inspection
Cost units
3
When counted
Decision is 1
Missed fault
Cost units
20
When counted
Fault is 1 and decision is 0
Model upkeep
Cost units
Separate budget
When counted
Only when comparing total ownership

We assume an inspection catches the fault in this teaching example.

Give a more complicated design a reason to exist

The two terms represent different consequences. Adding them makes the cost assumptions visible.

Which policy loses less on the same pumps?

Change the fault labels, decisions, or costs. Compare each policy under the same assumptions.

BOTIE

Botie needs a comparison that charges for inspections and missed faults on the same pump records.

Choose an example

Confirmed faults, 1 = faulty

Inspection decisions

Cost per inspection

Cost per missed fault

Policy loss in illustrative cost units

Follow the operations

1. Inspection decisions
1. 01
2. 11
3. 20
4. 30
2. Cost per inspection
```
3
```
3. Multiply
1. 03
2. 13
3. 20
4. 30
4. Confirmed faults, 1 = faulty
1. 01
2. 10
3. 21
4. 30
5. Set a constant
```
1
```
6. Subtract
1. 00
2. 10
3. 21
4. 31
7. Multiply
1. 00
2. 10
3. 21
4. 30
8. Cost per missed fault
```
20
```
9. Multiply
1. 00
2. 10
3. 220
4. 30
10. Add
1. 03
2. 13
3. 220
4. 30
11. Add a list
```
26
```
12. Result
```
26
```

Fault labels are supplied for all pumps. This omits uncertain inspection effectiveness, upkeep, and limited capacity.

1. Start with a rule we can explain

A baseline is a procedure we can run before adding sophistication. It might inspect pumps above a fixed temperature or show the most frequently requested help article. It gives us an answer to compare against and often exposes missing infrastructure early.

Run that rule through the complete path from input to action. Discovering that warnings reach nobody is useful before spending a week tuning a model.

Keep the baseline available throughout development. A candidate should earn its extra maintenance burden.

2. Put both policies on the same evidence

Use pumps from the same evaluation period, with fault outcomes obtained in a comparable way. Preserve a separate development set for choosing thresholds. Otherwise, repeatedly adjusting the model to beat the evaluation pumps turns them into training information.

The fault labels below are supplied for every pump. Real inspection data may lack that coverage. A fair trial might need some independently sampled inspections to learn about pumps the old policy ignored.

Missing outcomes weaken the comparison. They cannot be repaired simply by giving uninspected pumps a healthy label.

3. Charge for the action and the miss

Let a decision of 1 mean inspect. Subtracting it from 1 identifies a deferred pump. Multiplying that result by the fault label identifies a missed fault. An inspection has a fixed cost in this example, even when the pump is healthy.

We assume inspection prevents the whole stated miss loss. If inspection sometimes fails to help, the outcome model needs another term.

Labels = [1, 0, 1, 0]
Rule decisions = [1, 1, 0, 0]
Misses = labels × (1 − decisions) = [0, 0, 1, 0]
Rule loss = 3 × 2 inspections + 20 × 1 miss = 26
Model decisions = [1, 0, 1, 1]
Model loss = 3 × 3 inspections + 20 × 0 misses = 9

On these pumps and these costs, the model has lower decision loss. That statement still depends on the evidence and assumptions.

Find the misses before costing them

Boolean comparisons can show which faulty pumps a policy defers. These arrays align one position per pump.

Python

import numpy as np
fault = np.array([1, 0, 1])
inspect = np.array([0, 1, 1])
missed = (fault == 1) & (inspect == 0)
print(missed.tolist())

Output

[True, False, False]

The middle pump incurs an inspection, but it is not a missed fault.

4. Count the work needed to keep it running

Training a model once is only part of its cost. Someone must maintain input definitions, investigate failed requests, collect later labels, and check new versions. The heat rule also needs maintenance, so compare the additional burden fairly.

Use the same time horizon. A monthly operating estimate cannot be added directly to a loss measured over four pumps. First estimate how many comparable decisions occur in a month and how uncertain that extrapolation is.

A small measured improvement may disappear under a plausible operating cost. That is a reason to gather better evidence, not automatically a reason to abandon ML.

5. Some requirements stay outside the cost score

If a requirement says the team can perform only a certain number of inspections, a low cost score does not grant extra capacity. Check the hard limit separately. The same applies to a service that must respond before a machine’s next cycle.

A weighted score allows one quantity to compensate for another. Use it only where that trade is actually acceptable. We will make this distinction explicit when comparing several designs.

Inspect-all can be a useful reference policy, even if it is infeasible. It reveals how much of the apparent improvement comes from doing more work.

6. Choose the next experiment from the disagreement

Find the pumps where the baseline and candidate disagree. Are those cases genuinely different, or does the candidate rely on a sensor that is often unavailable? Examine which conditions create misses, then decide what additional observations would clarify them.

Report results by relevant operating condition as well as overall. A rule that works for one pump family may fail on another. Improvement on this tiny table is an arithmetic fact, not evidence of generalisation.

Botie will calculate the loss for several policies. Python will package that calculation so we can compare new aligned records without rewriting it.