Stop a Broken Sensor Batch Before It Reaches Training

Checking Data Before Fitting

BOTIE

A training program can finish successfully after a producer changes the meaning of a column. There need not be a syntax error or a crash. The model simply learns from the wrong observations, and its predictions may later feed more bad examples into training.

Breck and colleagues described this production problem in their 2019 paper on data validation at Google. Their system, part of TensorFlow Extended, compared incoming data with a schema and examined differences between datasets. A schema records expectations such as which fields exist and which values are allowed. Checking those expectations gives a pipeline a chance to identify a bad input before fitting another model.

We will make the decision visible on one sensor batch. First inspect readings against an allowed interval, then decide whether the rejected fraction requires stopping the batch. The example is small enough to check by hand, but the responsibility is the same: a completed calculation does not establish that its input was suitable.

Sources for this history

The New Firmware Sends Plausible-Looking Numbers

A sensor upgrade starts reporting values outside its documented range. Dropping those rows lets training continue, but it also removes nearly the entire night shift. The cleaned file looks tidy while the training population has changed.

We need two decisions: which rows meet the contract, and whether enough of this batch remains to use it. Keep the rejected rows so the producer can be repaired.

This checks finite numeric ranges. Release thresholds and producer diagnosis are developed in the Python challenge.

Step 1 of 1

Do not make the discarded rows disappear

The accepted interval is inclusive from 0 to 100.

Rejected records retain their identity and original value in a quarantine store.
Reading	Accepted?	Reason
20	Yes	Inside interval
-3	No	Below minimum
120	No	Above maximum

20
Accepted?
Yes
Reason
Inside interval
-3
Accepted?
No
Reason
Below minimum
120
Accepted?
No
Reason
Above maximum

Rejected records retain their identity and original value in a quarantine store.

A contract describes meaning

The original batch size remains visible after filtering.

Keep the rejected sensor evidence visible

A clean-looking file can conceal a broken producer.

BOTIE

A clean-looking file can conceal a broken producer.

Choose an example

Incoming readings

1

2

3

Minimum valid value

Maximum valid value

Accepted readings and rejected count

Follow the operations

1. Incoming readings
1. 00
2. 120
3. 2100
2. Minimum valid value
```
0
```
3. Threshold gate
1. 01
2. 11
3. 21
4. Maximum valid value
```
100
```
5. Threshold gate
1. 01
2. 11
3. 21
6. Multiply
1. 01
2. 11
3. 21
7. Select real entries
1. 00
2. 120
3. 2100
8. Count items
```
3
```
9. Add a list
```
3
```
10. Subtract
```
0
```
11. Collect two results
```
[
  [
    0,
    20,
    100
  ],
  0
]
```
12. Result
```
[
  [
    0,
    20,
    100
  ],
  0
]
```

This checks finite numeric ranges. Release thresholds and producer diagnosis are developed in the Python challenge.

1. A contract describes meaning

A field type answers whether a value is numeric. A range says which numerical values are valid under a specific measurement contract. A unit says what those values measure. None replaces the others.

Do not infer the contract from every incoming batch. A producer error would then redefine its own acceptance rule.

The exercise assumes finite numerical readings; parsing and missing-value policies belong before this range check.

2. Keep both branches visible

Compare each value with the lower and upper bounds. Combine the comparisons into a mask. Use that mask for accepted records, and its complement for rejected records.

readings = [20,-3,120]
accepted mask = [1,0,0]
accepted = [20]
rejected = 2
rejection fraction = 2 / 3

The workshop returns accepted values and a rejection count together so cleaning cannot hide its effect.

A mask preserves the rejected positions

NumPy comparisons return one boolean for each original reading.

Python

import numpy as np
x = np.array([20,-3,120])
keep = (x >= 0) & (x <= 100)
print(keep.tolist())
print(np.flatnonzero(~keep).tolist())

Output

[True, False, False]
[1, 2]

The rejected indices can be used to retain identities and reasons in a separate report.

Check yourself

Which decision belongs to the batch?

Question 1 of 2

How many rows pass?

Use the information below.

Allowed range [0,100], readings [20,-3,120].

3. Decide whether the batch is usable

A batch with one faulty reading may still meet a declared tolerance. A batch missing nearly every night observation may need to be held. Our Python policy holds a batch when its rejection fraction exceeds the supplied limit, and holds empty batches explicitly.

Inspect rejection rates by producer and relevant group. A small overall fraction can conceal a complete failure in a small population.

Passing a structural contract does not prove a dataset is unbiased, representative or correctly labelled.

4. Repair upstream and replay deliberately

Preserve the raw input version and reasons for rejection. When the producer is fixed, reprocess into a new version and compare accepted counts and model behaviour.

Quarantine must itself have retention and access controls. It is an investigation path, not an indefinite dumping ground for sensitive records.

A pipeline should make failure visible to its operator rather than silently training on whatever survived.