Threats Across the ML Lifecycle
Suppose a model usually works, but gives the wrong answer whenever an input contains a particular mark. Where did that behaviour come from? We’ll investigate how changes to training records can affect a model, and how someone using the finished model might find a different way to make it fail.
Protecting the training process
A training pipeline accepts a file. Who supplied it, and what could its contents make the model learn? We’ll keep the source of the records visible while testing a suspected change in behaviour.
- 01Ready to exploreWho Was Allowed to Teach the Model?
Trace proposed training records to reviewed sources before they influence a model, while keeping the limits of that evidence visible.
- 02Ready to exploreThe Model Passed Until a Particular Mark Appeared
Separate ordinary accuracy from a declared targeted-failure experiment, keeping eligibility and missing coverage beside the rate.
Assessing exposure through prediction
An attacker may change an input or call a prediction service many times. We need to state which actions are available before an experiment can tell us whether a defence helped.
- 03Ready to exploreWhich Changes Is the Opponent Actually Allowed to Make?
Define input control and a perturbation budget before interpreting a model failure as evidence about a particular threat.
- 04Ready to exploreA Prediction Service Can Reveal More Than One Answer
Count the work inside a request, minimise exposed prediction details and distinguish practical controls from a guarantee against model extraction.