When Should the Model Ask for Help?

Sending uncertain cases to a person changes both the error rate and the workload. Measure what the model handles and what it leaves for the review team.

13 sections · 3 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Allowing a Classifier to Decline a Decision

BOTIE

Selective classification studies predictions with a reject option: the system can decline some cases and trade coverage for lower error on accepted cases. Geifman and El-Yaniv developed a method for deep networks with risk control under their stated procedure.

Our threshold exercise measures empirical coverage and error on supplied cases. It does not implement that paper’s guarantee. The useful intuition is that refusing a case changes the population whose errors we report.

Sources for this history

Geifman and El-Yaniv: Selective Classification for Deep Neural Networks, 2017

The Review Desk Has Only So Much Time

Botie classifies pump inspections, but a person can review cases it is unsure about. That seems helpful until almost every inspection lands on the desk. We have reduced the model’s workload by giving it to someone else.

We need to measure two things together: how much work the model accepts and how often it is wrong on that accepted work. Then we can check whether the remaining queue is manageable.

Invented confidence scores and observed error flags. Confidence is a supplied selection score, not a verified probability of correctness.

Step 1 of 2

Acceptance changes the denominator

An error flag of 1 means the frozen model’s predicted class was wrong.

The review queue contains C even though the recorded prediction happened to be correct.
Inspection	Confidence	Error flag	Accepted at 0.75?
A	0.95	0	Yes
B	0.80	1	Yes
C	0.60	0	No

A
Confidence
0.95
Error flag
0
Accepted at 0.75?
Yes
B
Confidence
0.80
Error flag
1
Accepted at 0.75?
Yes
C
Confidence
0.60
Error flag
0
Accepted at 0.75?
No

The review queue contains C even though the recorded prediction happened to be correct.

Step 2 of 2

A confident mistake can survive a stricter rule

Now A is the error. Raising the threshold keeps that wrong prediction.

Confidence thresholding does not guarantee that the accepted error rate decreases on every sample.
Threshold	Accepted	Review	Accepted error rate
0.75	2	1	0.5
0.90	1	2	1

0.75
Accepted
2
Review
1
Accepted error rate
0.5
0.90
Accepted
1
Review
2
Accepted error rate
1

Confidence thresholding does not guarantee that the accepted error rate decreases on every sample.

The rejected cases still exist

The threshold tool is reused with a new, explicitly different score meaning. The report keeps the rejected work visible.

Watch both sides of the handoff

Threshold the supplied confidence, then count accepted errors and cases sent for review.

BOTIE

Botie can ask for help, but a handoff changes both the accepted sample and the review queue.

Choose an example

Confidence in predicted class

Observed error flags

Acceptance threshold

Coverage and risk

{
  "accepted": 2,
  "review": 1,
  "coverage": 0.6667,
  "risk": 0.5
}

Follow the operations

1. Confidence in predicted class
1. 00.95
2. 10.8
3. 20.6
2. Acceptance threshold
```
0.75
```
3. Accept scores at the threshold
1. 01
2. 11
3. 20
4. Observed error flags
1. 00
2. 11
3. 20

5. Report coverage and accepted errors

{
  "accepted": 2,
  "review": 1,
  "coverage": 0.6667,
  "risk": 0.5
}

6. Result

{
  "accepted": 2,
  "review": 1,
  "coverage": 0.6667,
  "risk": 0.5
}

Observed confidence is not a correctness guarantee. The fixture does not simulate reviewer outcomes or queues.

1. Separate the prediction from the decision to use it

The classifier produces a predicted label. A selector decides whether the application should accept that label or request review. These are distinct decisions. For a binary probability, confidence might be the larger class probability, but a different selection score can be trained or calibrated.

Our inputs already contain confidence and an observed error flag for each frozen prediction. A low positive-class probability is not necessarily low confidence: it may express confidence in the negative class.

Binary p(fault) = 0.05
Predicted class = healthy
Maximum class probability = max(0.05, 0.95) = 0.95

Do not reuse a positive-class alert threshold as a confidence threshold without changing its meaning.

2. Measure coverage and the errors left inside it

Coverage is the fraction of all cases accepted by the selector. Selective risk, for zero-one loss, is the error fraction among accepted cases. Rejected cases do not enter that error denominator, but they remain in the workload report.

If nothing is accepted, selective risk is undefined. Reporting zero would make a system that refuses every task look flawless.

At threshold 0.75:
Accepted = 2 of 3
Coverage = 2 / 3
Accepted errors = 1
Selective risk = 1 / 2

A lower accepted error rate can accompany a much larger review queue.

3. Test the confident mistakes

Raising the threshold changes which cases remain. It often removes difficult examples when the selection score is useful, but overconfident mistakes can stay. Examine them directly and report the coverage-risk tradeoff across development thresholds.

Calibration of probabilities and ranking of correctness are related questions with different answers. A system can order cases usefully while overstating their probabilities, or be well calibrated in broad bins while still failing in an important subgroup.

Freeze a chosen selector before assessing it on new cases. Repeatedly tuning against the final assessment reuses its evidence.

4. Treat review as part of the system

Count arrivals and service capacity over time, not only a daily total. Cases can arrive in a burst while the reviewer is unavailable. Decide what happens when the queue is full and how urgent cases are prioritised.

Human review is not automatically correct. Measure reviewer agreement, turnaround time and downstream outcomes. If the model’s label is shown to reviewers, consider whether it anchors their decisions.

100 cases arrive in a shift
Coverage = 0.70
Review arrivals = 100 × (1 - 0.70) = 30
Review capacity = 20
Unserved cases in this simplified shift = 10

This count ignores arrival timing and existing backlog. A deployment plan needs those details too.

5. Check who is being referred

Compare acceptance rates and accepted errors across relevant groups. A model could look better overall because it sends nearly every case from an underrepresented group to a slower manual route.

Report outcomes for all incoming cases where they can be obtained, including rejected ones. Labels collected only for accepted cases create a selective observation problem: the missing outcomes can hide failures in the selection policy.

A useful selector needs an evaluation of the complete service, not only its easiest accepted cases.

6. Use the mask on the errors, not on the denominator twice

The acceptance mask selects the observed error flags. Averaging those flags gives accepted risk. Averaging the mask gives coverage because its zeros and ones refer to every incoming case.

The challenge must handle the empty accepted set explicitly.

Inspect accepted cases

This example demonstrates the mask on a different three-case batch.

Python

import numpy as np
confidence = np.array([.9,.4,.8])
errors = np.array([0,1,1])
accepted = confidence >= .8
print(errors[accepted].tolist())
print(int((~accepted).sum()))

Output

[0, 1]
1

The rejected error remains part of the workload to resolve. It is excluded only from the accepted-error metric.