Investigate a Sensor Before Calling It Faulty

Botie cannot ask a technician to inspect every pressure reading. He needs a small, understandable alert procedure and an honest estimate of how often it sends the wrong cases for review.

We will build the alert procedure from the readings through to the decision to request an inspection. Along the way, we need a fitted reference and a way to select a threshold without using the final test answers. The records are small and synthetic so we can check those choices before trying the procedure on a larger dataset.

The reference and threshold have already been selected. Apply them to the supplied readings without relearning either quantity from the test window.

The outcome is whether a reading corresponds to a confirmed fault under the inspection procedure. The action is sending a case for review. For this project, a false alert costs one unit and a missed fault costs five units.

These are invented decision costs. In practice they should reflect the burden and consequences of investigation. Keep measurement units and operating mode consistent across the three windows.

A score alone does not choose the right operational threshold.

Use only the earlier reference values to fit median and raw MAD. Retain both values. The no-alert baseline misses all faults but creates no investigations, so it provides a useful cost comparison.

Check that the MAD is positive. If it is zero, stop and choose a documented reference policy before proceeding. Do not replace it using test data.

Reference median = 10; raw MAD = 1
Validation scores for [10,13,15] = [0,3,5]

The raw MAD convention must remain identical in every window.

Evaluate thresholds two and four on validation, using a strict greater-than comparison. Count false alerts and missed faults separately, multiply by their costs, and choose the smaller total. Use the first threshold on a tie.

Do not add more threshold candidates after inspecting the test outcomes. That would turn the final test into another selection set.

Threshold 2: one false alert, zero missed faults; cost = 1
Threshold 4: zero false alerts, zero missed faults; cost = 0
Selected threshold = 4

A perfect result on three validation records is weak evidence; it is only enough to illustrate selection.

import numpy as np
readings = np.array([8., 10., 16.])
saved_centre, saved_mad = 10.0, 2.0
threshold = 2.0
scores = np.abs(readings - saved_centre) / saved_mad
print(scores.tolist())
print((scores > threshold).tolist())

[1.0, 0.0, 3.0]
[False, False, True]

Score the later test values using the same centre and MAD. Apply the selected threshold once, then compute alerts and costs from the held-out labels. Preserve per-record results so aggregate performance can be inspected.

Report the tiny sample size, operating conditions, and synthetic nature of this exercise. Do not treat its error rate as a production guarantee.

Test scores = [1,4,6]
With score > 4: alerts = [False,False,True]
Test cost = 0

The reading exactly at the threshold does not alert.

For each alert, retain the original reading, reference version, score, threshold, and eventual inspection result. Later revisions can then distinguish a model change from a changed label or sensor repair.

Inspect missed cases too. A review process that labels only alerts cannot directly measure how many unalerted faults occurred without additional sampling.

Your deliverable is a reproducible procedure and an evaluation record, not simply a fitted detector.

A detector produces an alert before a person knows whether the sensor failed. Give every reading a stable identifier and store its value, score, threshold, rule version and decision. Add the inspection result later, rather than overwriting the original decision.

An uninspected reading does not mean a healthy sensor. If inspectors only examine alerts, measured precision may be available while missed failures remain unknown. Sample some non-alerts for inspection when the project needs a recall estimate. Keep these two sources of evidence separate in the report.

The reference median and median absolute deviation describe the period we chose as normal. A new extreme reading must not change those numbers while it is being scored. Otherwise, the reading can move its own threshold.

A changed production process may eventually justify a new reference period, but inspect the shift first. Store the new rule with a new version and compare its decisions with the previous one. A zero reference deviation needs an explicit policy; our function rejects it instead of dividing by zero or inventing a tolerance.

Save the frozen rule with the values, scores and alert flags. A second program should be able to reproduce every decision using only that report. Include the strict comparison convention: a score equal to the threshold is not an alert here.

Try a fresh batch with one enormous reading. Scores for the unchanged readings should stay the same. Then try a genuinely different reference rule and explain which decisions change. The report is an inspectable record of a rule, not proof that every alert was a real fault.