When the Important Class Is Rare

If almost every sensor is healthy, predicting healthy every time can look accurate while missing every failure. We need to inspect what the score says about each class.

12 sections · 3 questions · 2 coding challenges

1 / 12

The arrow keys move between sections too.

Learning From a Class That Appears Only Occasionally

BOTIE

Consider a classifier trained on records where one outcome is much more common than the other. It can reduce its total mistakes by favouring the common class, even if the rare cases are the ones we most need to find. Work on imbalanced classification addresses that mismatch between a convenient training objective and the job the model is meant to do.

Chawla and colleagues' 2002 SMOTE paper studied synthetic minority oversampling, creating additional minority examples between nearby observations. That is one response to imbalance, not a universal remedy. Our lesson first makes the evaluation problem visible; it does not generate synthetic sensor records.

Sources for this history

Chawla and colleagues: SMOTE (2002)

How Did a Model Miss Every Failure and Still Look Good?

Botie checks a batch containing 95 healthy sensors and five failed sensors. A rule that always says healthy gets most answers right. It also fails at the entire reason we might have built an alert system.

We need to ask how well the rule finds the five failures. That is a different question from how often it is right across the whole batch. The class counts affect training as well as evaluation, so we will also look at what changes when mistakes on the rare class receive more weight.

The class counts describe an illustrative evaluation, not a deployed detector.

Step 1 of 3

Count both classes

The data contain many more healthy sensors than failed ones.

Healthy sensors95

Failed sensors5

Most recorded sensors are healthy. A rule can therefore appear accurate while missing every failure.

Step 2 of 3

Inspect the always-healthy rule

The rule gets all healthy sensors right and misses all failures.

Check the rule separately on each known class, rather than letting the larger group hide the other one.
Known class	Available examples	Correctly identified
Healthy	95	95
Failed	5	0

Healthy
Available examples
95
Correctly identified
95
Failed
Available examples
5
Correctly identified
0

Check the rule separately on each known class, rather than letting the larger group hide the other one.

Step 3 of 3

Give each class a visible score

The class-specific recalls reveal the failure that overall accuracy conceals.

Healthy recall = 95 / 95 = 1.0
Failure recall = 0 / 5 = 0.0

Recall asks how much of each actual class was recovered.

Each class needs its own account of the mistakes

Division makes class-specific recall. Averaging those recalls gives each class equal weight in the summary.

Did the model find the failures?

A model can look accurate by mostly recognising the common class. Change the results for one class and inspect its recall before averaging across classes.

BOTIE

Botie needs a score that gives each sensor class a place. Compute each class's recall before combining the results.

Choose an example

Correct predictions per class

Total observations per class

Mean of the class recalls

0.5

Follow the operations

1. Correct predictions per class
1. 095
2. 10
2. Total observations per class
1. 095
2. 15
3. Divide
1. 01
2. 10
4. Mean
```
0.5
```
5. Result
```
0.5
```

All supplied class counts are positive. Balanced accuracy does not encode application-specific mistake costs.

1. Count the Mistakes the Task Cares About

Accuracy divides all correct predictions by all examples. If one class dominates the dataset, it also dominates that average. The problem is not that the arithmetic is wrong; the score answers a question that may be insufficient for our task.

Always-healthy accuracy = 95 / 100 = 0.95
Failure recall = 0 / 5 = 0

A high overall score does not establish that rare events are being found. Start by inspecting the class counts and the confusion matrix.

2. Give Each Class an Equal Place in the Summary

Recall asks what fraction of a class was correctly identified. Balanced accuracy averages those recalls, giving each class equal weight in the summary regardless of its count.

For the always-healthy rule, the healthy recall is perfect and the failure recall is zero.

Balanced accuracy = (1 + 0) / 2 = 0.5

This makes the neglected class visible. It still does not specify the cost of a false alarm or a missed failure, and it does not replace the individual class scores.

3. Separate Training Choices From Evaluation Choices

We can weight training losses, resample training examples, or change the decision threshold. These are different interventions. Loss weighting changes the fitting objective; resampling changes the examples presented; a threshold changes how scores become actions.

Fit these choices using training and validation data. Oversampling before splitting can place related or duplicated information into both training and evaluation sets.

A balanced training sample does not imply balanced deployment data. Keep the final evaluation distribution appropriate to the real task, and report the class counts used.

4. More Copies Are Not More Independent Evidence

Repeating a rare observation can make it influence training more, but it does not create a new independent observation. Synthetic examples introduce additional assumptions about which points are plausible.

For a rare class, score estimates can also be unstable because they depend on few cases. Report that limitation and inspect which mistakes occurred. The workshop constructs balanced accuracy from separate correct and total counts for both classes.

An evaluation set with no examples of a required class cannot estimate that class's recall. Do not quietly replace an undefined recall with a reassuring value.

5. More Copies of a Failure Do Not Add New Failures

Oversampling can make rare failures appear more often during fitting. It changes the training emphasis; copying one faulty sensor repeatedly does not provide new independent evidence about faults. Split by the appropriate observation or group before resampling.

Weighting the loss is another way to change emphasis. Neither choice guarantees useful probabilities at the original failure rate. Evaluate on a population that matches intended use, and check calibration as well as how many failures the model detects.

6. Put the Inspection Budget Beside Recall

Botie can inspect only a limited number of sensors each morning. A model that finds more failures by calling nearly everything faulty may not fit that budget. Examine true failures found, unnecessary inspections and missed failures together.

Choose the rule using validation data, then freeze it for the held-out assessment. If labels exist only for inspected sensors, acknowledge that missed failures among uninspected sensors are not fully observed.

Count the consequences of the chosen calls

Labels are 1 for faulty and 0 for healthy. These are illustrative held-out decisions.

Python

import numpy as np
faulty = np.array([1, 0, 0, 1, 0, 0])
inspect = np.array([True, True, False, False, False, True])
print(int(inspect.sum()))
print(int(((faulty == 1) & inspect).sum()))
print(int(((faulty == 1) & ~inspect).sum()))

Output

3
1
1

Three inspections find one of the two failures. Reporting only overall accuracy would hide the operational question.