Finding Unusual Readings Without Letting Them Set the Reference

Most of Botie’s pressure readings are close together, but one is much larger. If that reading helps determine the average and spread used to judge it, it can make itself look less unusual.

Can we choose a reference that one extreme reading has less power to move? The median gives us a centre, and the median absolute deviation gives us a measure of spread. We can use them to judge how unusual a reading is relative to the reference records. That score will not tell us why the reading is unusual; a fault is one possibility to investigate.

Reference median: 10
Reference MAD: 1
New reading: 13
Raw MAD score = |13 − 10| / 1 = 3

Fit the centre and spread from the reference records before scoring incoming values. Introduce an extreme reference value and compare its influence on those summaries.

Choose records that represent the operating condition in which the detector will be used. A pressure normal during startup may be unusual during steady operation. Split reference fitting from later threshold selection and final testing.

An anomaly score measures disagreement with the reference. It does not tell us whether an observation is a sensor fault, a genuine event, or a new operating regime.

The same numerical deviation can have different meanings in different conditions.

The median is a central ordered value. The median absolute deviation, or MAD, is the median of distances from that centre. Neither step requires squaring a large deviation.

Reference = [8,9,10,11,100]
Median = 10
Absolute deviations = [2,1,0,1,90]
MAD = 1

The centre and spread should be retained for scoring later records.

Our raw score divides absolute distance from the median by MAD. A commonly used normal-consistency scaling multiplies MAD by approximately 1.4826; that produces a different numerical score and threshold.

We use the raw convention throughout this lesson. If MAD is zero, many reference values coincide. Do not silently divide by zero or insert an unexplained epsilon: define a fallback scale, exact-match policy, or a different model.

Raw score for 13 = abs(13-10) / 1 = 3

A threshold has meaning only together with the score definition and reference data.

import numpy as np
reference = np.array([4., 5., 6., 7., 8.])
centre = np.median(reference)
departures = np.abs(reference - centre)
print(float(centre))
print(departures.tolist())
print(float(np.median(departures)))

6.0
[2.0, 1.0, 0.0, 1.0, 2.0]
1.0

Select a threshold using representative validation records and the cost of missed faults versus unnecessary investigations. Evaluate precision and recall when reliable labels exist, and inspect the burden of alerts per day.

Updating the reference automatically with every incoming reading can gradually absorb a developing fault. Decide which records are eligible to update the baseline and keep versions for comparison.

Botie’s workshop fits a median reference, then scores separate new readings.

Several repeated pressure readings can give a median absolute deviation of zero. Dividing by it is undefined. Adding a tiny constant merely chooses an extremely sensitive scale, often without a physical justification.

Declare a fallback based on measurement resolution or another fitted scale, or report that this score cannot be formed. Keep the median and scale fitted on the reference period when scoring new batches. Re-estimating them on each incoming batch can hide a whole shifted batch.

Pressure and flow may usually rise together. A new observation can be ordinary on both individual ranges while contradicting their relationship. Separate univariate scores will miss that kind of anomaly.

A residual from an explicitly fitted relationship provides one way to inspect it. This example supplies a simple reference relationship so that we can distinguish marginal range checks from joint behaviour.

import numpy as np
reference = np.array([[1.,1.],[2.,2.],[3.,3.]])
new = np.array([1.,3.])
inside = (new >= reference.min(axis=0)) & (new <= reference.max(axis=0))
print(inside.tolist())
print(float(new[1]-new[0]))

[True, True]
2.0

An unusually large score is a reason to investigate, not a fault label. Choose the inspection threshold or review budget using a suitable validation period and whatever outcome labels are available.

Record unresolved cases separately from confirmed healthy cases. If only high-scoring records are inspected, the labelled subset cannot establish how many low-scoring faults were missed. A sampling plan for some lower-scoring records can help assess that blind spot.