Which Changes Is the Opponent Actually Allowed to Make?

Botie adds small random changes to sensor readings and usually gets the same correct prediction. That tells us something about those random trials. Now suppose someone can choose a change after seeing how the model responds. They may find a wrong decision that the random trials never reached.

Before building an experiment, we need an agreement about the opponent: which readings can they affect, how much may they change them, and which wrong outcome are they trying to produce? That agreement lets us distinguish a relevant failure from a dramatic-looking example outside the stated threat.

A wrong changed prediction matters to this experiment only if the opponent was allowed to make that change. Botie needs both the model result and the capability checks.

{
  "within_scope": true,
  "clean_correct": true,
  "changed_correct": true,
  "observed_failure": false
}

At prediction time, an opponent can try to change the input so the model makes an unwanted decision. This is called evasion. Poisoning acts earlier, through material used during learning. An evasion goal may ask for one particular wrong class, called targeted, or accept any wrong class, called untargeted.

Our workshop represents a sensor observation with two numerical entries between 0 and 1. Each entry is a coordinate of that vector. The reviewed true class stays 0, and the unwanted prediction is 1. The opponent may change the first coordinate only. That control is an assumption of the scenario, not a property supplied by the classifier.

A useful threat model ties control to the actual system: who can supply a request, alter a file or influence a physical reading?

An evaluator may know the model’s weights, obtain only query responses, or have partial information. White-box and black-box are common names for those different knowledge/access assumptions. Neither term specifies which input changes are allowed.

Record query or computation budgets as well. An experiment with a few random probes and one with extensive adaptive search provide different evidence even under the same input-change limit.

The local workshop exposes weights so the boundary is easy to understand. It does not emulate an unknown remote model.

We need a precise meaning for small. One option is to calculate the absolute change in each coordinate and require even the largest change to stay within a chosen limit. This is an L-infinity budget. A separate list of 0-or-1 flags identifies which coordinates the opponent may change at all.

Other rules can bound total Euclidean distance or the number of edited entries. A physical scenario might instead specify a camera angle or a possible sensor intervention. Those rules allow different inputs, so the same numerical limit cannot simply be moved from one to another.

original = [0.375,0.5]
changed = [0.5,0.5]
absolute changes = [0.125,0]
largest coordinate change = 0.125
permitted largest change = 0.125
mutable coordinates = [1,0]
result = within the stated change budget and coordinate policy

A domain-range check remains necessary even when the change size is small.

import numpy as np
original=np.array([.375,.5])
changed=np.array([.5,.5])
print(np.abs(changed-original).tolist())
print(float(np.max(np.abs(changed-original))))

[0.125, 0.0]
0.125

First ask whether the changed vector satisfies the domain, fixed-coordinate and size rules. Separately calculate the original and changed predictions. Combining these results gives a meaningful report.

A changed prediction can be wrong while the trial is out of scope. Keep that fact as a diagnostic observation, but do not count it as a failure under a narrower capability claim. An original prediction that was already wrong is also a different issue from a newly induced failure.

weights = [1,1]
cutoff = 1
original score = 0.375 + 0.5 = 0.875
original prediction = 0
changed score = 0.5 + 0.5 = 1
changed prediction = 1
reviewed true label = 0
observed new within-scope failure = true

The equality convention at the cutoff is part of this model’s contract: a score reaching the cutoff predicts 1.

A mathematical budget does not decide what an input means. A small edit can remove essential evidence, and a visually large edit can sometimes preserve a label. Define a task-appropriate review process before assuming that the original label remains correct.

For the workshop’s invented sensor scenario, the supplied labels are reviewed fixtures. In an image, speech or physical-control application, label preservation and physical feasibility need their own evidence.

The threat model should expose this assumption instead of hiding it behind the word small.

Finding one valid failure establishes a counterexample to the tested robustness claim. Failing to find one with a finite search does not prove none exists. Report the search method, budget, examples, exclusions and ordinary accuracy.

Random noise is useful for its own reliability question. It cannot stand in for an opponent adapting inputs to the model. A defence also needs evaluation against the actual defended system, rather than assuming that making one search method fail removes the underlying weakness.

An optimisation failure and a robust model can look alike to a weak test. That is why evaluation methods and assumptions belong beside the result.

A system might reject suspicious inputs, change preprocessing, retrain with justified examples or route uncertain cases for review. Each choice can change ordinary accuracy, coverage and resource cost. Re-evaluate the declared threat and normal operating data after a repair.

An always-reject system prevents a model prediction by refusing service, which may be unacceptable for the application. Make that tradeoff visible instead of reporting only a reduced failure count.

The appropriate response depends on the consequence of the decision and the real input channel.

After observing one boundary crossing, test another valid input and also an invalid capability case. A repair that hard-codes one vector can pass the incident while failing a nearby request. A report that ignores the fixed-coordinate rule can count an impressive but irrelevant failure.

Preserve the original incident with its model version, representation and scope policy. A later change of normalisation can change what the same numerical budget means.

Keep the original correctness check visible so ordinary model errors cannot disappear behind the newly induced-failure definition.

The report below shows whether both predictions were correct, even for an excluded trial. Its observed_failure field is null when the trial is out of scope or the original prediction was already wrong. Within scope, a previously correct example produces true or false according to the changed prediction.

This prevents an excluded failure from becoming a claimed in-scope counterexample and prevents an untested comparison from being labelled secure.

Keep the underlying scope and correctness fields with the summary so a reviewer can understand why an outcome was excluded.