Who Was Allowed to Teach the Model?

Botie uses the average of recorded maintenance targets as a simple starting prediction. Now a supplier proposes another set of targets, including a very large value. If that record enters the average, Botie’s prediction moves toward it. The fitting code can work exactly as written while learning from a contribution that nobody authorised.

We need to ask who could contribute records, which labels were reviewed and which release used them. That gives us a boundary to defend before fitting begins. Unusual data is not automatically malicious, and a familiar source is not automatically trustworthy forever.

A new feed supplies numerical targets, but this release has not approved its source. Botie needs to keep it out of the fit while retaining the accepted records’ identities.

[
  [
    "a",
    "b"
  ],
  2
]

A poisoning attempt changes something used during learning, such as training examples, labels or an update supplied to a collaborative training system. Its goal might be broadly worse predictions or a particular wrong decision. The attacker’s available channel matters as much as the model type.

For our maintenance baseline, the threat is an unapproved feedback feed entering the training set. This is different from a legitimate new operating condition whose measurements deserve investigation.

A detector that flags unusual values can identify review candidates. It cannot infer intent from unusualness alone.

If a target later turns out to be wrong, we need to find where it came from and which models used it. That record of origin and handling is provenance. Give a record an ID that stays attached to it, then retain its collection source and review status through the transformations that follow. A dataset release should name the records it included.

Evidence about the source must come from outside the record’s own claim. Depending on the system, that may involve authenticating the producer or checking a trusted signed manifest. A matching file hash confirms expected bytes; it cannot establish that a label inside those bytes is correct.

The workshop receives source IDs and review flags already resolved by a trusted ingestion process. It does not implement that process or verify signatures.

This release accepts a record only when its source is on the approved list and its label review is complete. Use the same eligibility mask for IDs and numerical targets. Filtering values without their identities would lose the evidence we need to reproduce the model.

proposed targets = [1,3,100]
approved-source flags = [1,1,0]
completed-review flags = [1,1,1]
accepted flags = [1,1,0]
accepted targets = [1,3]
fitted baseline = (1 + 3) / 2 = 2

The calculation shows the consequence of this policy. It does not establish that every approved source is honest.

rows = [{"id":"a","source":"field"},{"id":"x","source":"proposal"}]
approved = {"field"}
print([row["id"] for row in rows if row["source"] in approved])

['a']

A compromised approved source can pass an allowlist. A review process can also make systematic mistakes. Separate source authentication from label evidence and examine disagreements, repeated examples and changes in class or target distributions.

Duplicates can give one contribution more influence than expected. Group records by their actual contributing source or subject where the learning policy needs contribution limits. A count of distinct row IDs does not establish independent evidence.

There is no universal clipping threshold or outlier filter that makes a poisoned dataset safe. Strong screening also risks discarding legitimate rare examples.

Suppose the new supplier provides the training targets and the answers used to evaluate the model. A good score could mean that the model learned the supplier’s mistaken or manipulated convention. Reserve evaluation evidence with an appropriately independent collection and review process.

Look at relevant subsets as well as the total score. These subsets are often called slices: records from one source, for example, or one operating condition. A small affected group can disappear inside an average over many unaffected records.

An independent test set is useful evidence. It is not a guarantee against threats outside its coverage.

Keep unapproved or unresolved contributions outside the training release while they are reviewed. If the filter leaves no accepted records, report that there is no fitted mean. Returning zero would make missing evidence look like a measured baseline.

Keep the reason for exclusion and the policy version so a later review can reproduce the decision. Avoid copying sensitive raw examples into broad operational logs.

The workshop represents the no-evidence result with null. The service consuming that result needs an explicit stop or review path.

Stopping the source does not remove influence already learned by a deployed model. Use lineage to identify affected training runs, evaluation artifacts and deployments. Rebuild from a reviewed dataset, compare against an independent baseline and roll out the replacement through the normal release process.

Do not assume deleting a row retroactively unlearns its effect. Retain the incident fixture and ingestion checks so the same contribution path is tested before the next release.

The replacement release should identify the reviewed dataset, evaluation evidence and fitted artifact used in the repair.