← All workshopsRead Checking the Model After It Meets New Data

Deploying and Maintaining Models

Measure the error we can now observe

BOTIE

Subtract outcomes, take absolute values, and average over the monitored window.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Subtract outcomes, take absolute values, and average over the monitored window.

Botie's stored predictions now have matching later outcomes. This window can measure observed error rather than infer it from input drift.

Pair each prediction with its later outcome, take absolute differences, and average across the monitored window.

Examples and results
Recorded predictions
  1. 010
  2. 112
Matched later outcomes
  1. 011
  2. 115
Required result
2
Recorded predictions
  1. 04
  2. 15
Matched later outcomes
  1. 04
  2. 15
Required result
0
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

Labels must be correctly matched and representative. Report window size, model version, and subgroup results. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Delayed outcomes reveal an uneven error pattern

This check uses the pieces from “Measure error after outcomes become available”.

BOTIE

Outcomes have now arrived for four production predictions. Match each prediction to its outcome by row and calculate the mean absolute error. Botie will use this measurement alongside the deployment's agreed rollback policy.
Recorded predictions
  1. 020
  2. 120
  3. 220
  4. 320
Matched later outcomes
  1. 018
  2. 120
  3. 228
  4. 314

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

How could delayed or selectively observed labels bias this monitoring window?

Record window size, matching rules, model version, and subgroup coverage before acting on an aggregate error.

Labels must be correctly matched and representative. Report window size, model version, and subgroup results.

Work through the Python challenges →
Continue the courseWhat Makes This an Agent? →
← Knowing Which Model Produced an Answer