← All workshopsRead Does Eighty Percent Mean Eighty Percent?

Choosing and Checking a Model

Audit one group of failure probabilities

When the group averages agree

Botie's failure forecast says 80 percent for every sensor in a group. The observed outcomes give us a way to audit what that confidence meant.

BOTIE

The lower-risk group's mean probability equals its observed failure frequency. What should happen to its Brier score?
The inputs we are comparing

Failure probabilities in one group

Confident group

  1. 00.8
  2. 10.8
  3. 20.8
  4. 30.8

Lower-risk group

  1. 00.25
  2. 10.25
  3. 20.25
  4. 30.25

Observed failures

Confident group

  1. 01
  2. 10
  3. 21
  4. 30

Lower-risk group

  1. 00
  2. 11
  3. 20
  4. 30
Result for confident group
[
  [
    0.8,
    0.5
  ],
  0.34
]
Choose an explanation to test
The group agrees, but individual sensors still fail

This check uses the pieces from “Keep confidence, frequency and error separate”.

BOTIE

Five sensors each received a failure probability of 0.6. Return the pair of group averages, followed by the Brier score. Does agreement between the averages mean every forecast was correct?
Failure probabilities in one group
  1. 00.6
  2. 10.6
  3. 20.6
  4. 30.6
  5. 40.6
Observed failures
  1. 01
  2. 11
  3. 20
  4. 31
  5. 40

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What did the group-average comparison reveal that the Brier score alone did not?

Keep probability reliability separate from the threshold used to act on a forecast.

These tiny groups illustrate the arithmetic. They do not estimate calibration precisely or fit a correction.

Work through the Python challenges →
Continue the courseWhat Changed After the Model Was Fitted? →
← When Should a Probability Become an Action?