The Model Passed Until a Particular Mark Appeared

Botie’s image classifier recognises faults in the ordinary inspection set. A reviewer adds a small mark to paired copies of those images. The fault is still visible, but several predictions now say safe. The score on the original images never tested that change.

Keep each marked image beside its original. If the correct answer is still fault, did adding the mark cause a previously correct prediction to become safe? First set aside examples that were already wrong or were genuinely safe. They cannot show that this change induced a new wrong safe prediction. We will keep those counts visible while investigating the condition.

Botie’s ordinary validation never exposed a particular conditional failure.

[
  1,
  {
    "eligible": 2,
    "successes": 0,
    "rate": 0
  }
]

A backdoor is a learned conditional behaviour associated with a trigger under an attacker’s control. A particular input failure is not enough to establish that origin: ordinary model weaknesses, distribution changes or a label-changing edit can also cause failures.

State what your experiment observes. The fixtures below test a declared target condition and retain paired predictions; they do not tell us who trained the model or how it was trained.

Keep training provenance, model artifact identity and evaluation evidence available for an investigation without claiming causation from one score.

A mark may be irrelevant to an inspection label, but an edit that hides the actual fault may change what the image means. Have an appropriate review process define label-preserving changes before counting targeted failures.

Pair each changed input with its original example identity. Preserve that pairing through sorting, filtering and batching so the clean prediction belongs to the same object.

The reviewed-preservation flags are fixture evidence. The browser cannot decide visual or physical label validity on its own.

Count only pairs that can answer our question. The original true label must differ from the unwanted target, the original prediction must be correct, and review must establish that the change leaves the true label intact. These are the eligible pairs. Their count is the denominator, the group against which we measure the result.

Within that group, count changed predictions that reach the unwanted target. The output calls them successes from the attacker’s perspective; each is a failure of the model’s task. Other studies may choose another denominator, so publish this definition and the raw counts with the rate.

true labels = [fault,fault,safe]
clean predictions = [fault,fault,safe]
changed predictions = [safe,safe,safe]
target = safe
eligible flags = [1,1,0]
targeted successes among eligible = 2
eligible pairs = 2
conditional targeted-failure rate = 2 / 2 = 1

The ordinary clean accuracy is also 1. Those two measurements describe different questions.

truth=["fault","safe","fault"]
target="safe"
print([int(label!=target) for label in truth])

[1, 0, 1]

Conditioning on clean correctness makes the induced-change question interpretable, but it excludes examples the model already mishandled. Report clean accuracy and eligible counts alongside the conditional rate. A model with poor clean accuracy can otherwise be assessed on a small, unrepresentative subset.

If no pair is eligible, the conditional rate is undefined. Use null or an explicit unavailable label with the zero denominator. Calling that rate zero would claim evidence the experiment did not collect.

A small number of eligible pairs also leaves substantial uncertainty even when the observed rate is extreme.

Try the reviewed condition on examples from the classes and operating conditions that matter to the application. Also try control edits: changes chosen to check whether any image damage causes the failure, rather than something specific about the suspected mark.

Keep a separate set of reviewed pairs for evaluating a repair. If the developer repeatedly tunes the model on the same marked images, their improved score no longer supplies an independent check of the repair.

Passing one family of marks cannot establish the absence of every backdoor or every conditional failure.

A reproducible targeted failure can justify holding a release, investigating provenance and testing a replacement. The decision should include the application’s consequence and uncertainty, not only a clean benchmark threshold.

Screening methods can have false alarms and blind spots. Keep the tested threat conditions and method limitations with the release evidence rather than treating a single detector as a universal certificate.

This exercise reports observations. It does not provide a backdoor detector or a production security approval.

After a repair, repeat the original clean evaluation, the declared conditional experiment and a fresh reviewed pair set. A model that refuses every input can reduce targeted successes while destroying useful task performance.

Compare raw counts and eligible populations before comparing rates. Changing who qualifies can change the denominator without repairing the behaviour on the original examples.

Preserve the original incident pairs so a later dataset change cannot make the regression disappear from the report.

Use aligned label and prediction lists, with a preservation flag for every pair. Reject mismatched lengths instead of allowing zip to truncate them. Derive one eligible mask and reuse it for both the denominator and targeted-success count.

The challenge prints all reported fields. A caller can then see whether a low rate came from observed failures, an empty population or a narrower set of reviewed examples.

The class labels and prediction outputs are supplied test data. No attack is generated and no external model is queried.