Knowing Which Model Produced an Answer

Botie replaces a model and later receives a report about an incorrect prediction. The current model behaves differently, and the original files were overwritten.

To investigate the report, we need the exact version that made the prediction. Save each version's fitted state and a manifest describing its inputs and origin. Keep a reference to the version used for each prediction so we can reproduce the calculation or restore an earlier compatible procedure.

Version A = (3 × 2) + (2 × −1) = 4
Version B = (3 × 1) + (2 × 1) = 5

Both versions saw the same ordered fixtures. Change a prediction and inspect the largest difference before deciding whether the behavioural change is acceptable.

0.5

A human-readable version can identify a release, but a mutable file with that name can still change. A content hash detects byte changes in the serialized artifact.

A hash is not proof that the artifact is correct, safe, or from an authorised source. Provenance and integrity solve related but different questions.

Canonical serialisation makes equivalent manifest content hash consistently in this exercise.

A manifest can include feature schema version, preprocessing artifact, model artifact, training-data snapshot identifier, code revision, dependencies, seed, and evaluation results. Use references appropriate to the data’s privacy requirements rather than dumping training records into logs.

The manifest should distinguish the current candidate from the currently served version. Promotion changes which version is active without destroying prior artifacts.

A saved weight vector without its preparation and feature schema is an incomplete release.

Run candidate and current procedures on a fixed fixture set and compare predictions, validation metrics, numerical finiteness, and interface compatibility. A changed prediction is not automatically an improvement or a bug.

Track why the change was intended and inspect important boundary cases. A schema-incompatible artifact should not be promoted behind an unchanged request contract.

Fixture x = [3,2]
Version A: dot(x,[2,-1]) = 4
Version B: dot(x,[1,1]) = 5
Prediction difference B-A = 1

The numerical comparison needs labels or another criterion before choosing the better version.

import numpy as np
old_predictions = np.array([1., 2., 3.])
new_predictions = np.array([1., 2.5, 2.])
change = new_predictions - old_predictions
print(change.tolist())
print(float(np.abs(change).max()))

[0.0, 0.5, -1.0]
1.0

Keep the previous model together with its compatible preprocessing and runtime requirements. Reverting weights while keeping new preparation may create a third, untested procedure.

Record the active model identifier with predictions and monitoring summaries. Retention and access controls should reflect the sensitivity of any associated data.

The workshop compares version outputs; rollout and rollback decisions require the surrounding operational checks.

Changing feature order, category vocabulary or target units can make a model incompatible with an earlier caller even when its array shape stays the same. Version the whole fitted procedure and its contract together.

A release manifest should identify the artifact, training data snapshot or reproducible reference, code version, dependencies, evaluation report and intended use. A human-friendly version name helps discussion; a content hash helps identify the exact bytes.

A new model may legitimately change predictions. The comparison should make those changes visible and check both expected improvements and unacceptable regressions.

Use fixtures with normal inputs, boundary values and known failure cases. Compare target units and preprocessing before comparing numerical differences.

import numpy as np
old = np.array([10.,20.,30.])
new = np.array([10.5,19.,36.])
change = new-old
print(change.tolist())
print(np.flatnonzero(np.abs(change) > 2).tolist())

[0.5, -1.0, 6.0]
[2]

Do not overwrite the previous artifact when a candidate passes offline evaluation. Store releases immutably and change a separate active-version reference. Rollback then selects a known artifact rather than trying to recreate yesterday's model from memory.

Test that the earlier runtime and preprocessing still load correctly. A retained weight file is not a usable rollback if its dependencies or input contract have been removed.