One Correct Prediction Did Not Test the Interface

Write What Must Hold Across Many Inputs

BOTIE

A hand-calculated example can tell us what one prediction should be. It cannot list every future request. Some promises are easier to describe as relationships: if the same named measurements arrive in another order, the result should stay the same. Property-based testing starts from such a promise and tries many inputs that could contradict it.

Claessen and Hughes first published QuickCheck in 2000. The tool generated inputs for developer-written properties and helped reduce failures to smaller examples. Hypothesis brings this approach to Python. The useful result is a counterexample: a concrete input that breaks the stated promise. For our model interface, we will begin with the saved-centre mistake, then test whether changing feature order exposes another mistake. Each relationship must follow from the model’s actual behaviour.

Sources for this history

The Test Passed Until Another Pump Joined the Batch

Botie tests a centring function on a batch whose mean happens to equal the saved training mean. It returns the expected values. The same pump moves into a busier batch, and its centred value changes because the helper recalculates the mean from current requests.

We could save this one bad batch as a test. We should. But we can also ask a more useful question: if the fitted transformation processes examples independently, should adding another example change the first result? That relationship gives us a way to look for a whole class of mistakes.

This relationship belongs to a deterministic transform that processes examples independently. A batch-dependent model needs a different promise, and a passing relationship test still needs independently calculated result checks.

Step 1 of 1

The same pump appears in two request batches

The stored training centre is 4 in both requests.

The final row is the failure we want to expose, not an alternative fitted transformation.
Request	Input readings	First centred reading
Original batch	[2,6]	-2
Another pump joins	[2,100]	-2
Incorrect request-local centring	[2,100]	-49

Original batch
Input readings
[2,6]
First centred reading
-2
Another pump joins
Input readings
[2,100]
First centred reading
-2
Incorrect request-local centring
Input readings
[2,100]
First centred reading
-49

The final row is the failure we want to expose, not an alternative fitted transformation.

Begin with a claim the interface actually makes

A known expected result and a changed-batch relationship expose request-local relearning.

Keep one pump’s transformed value stable across batches

Botie’s first test accidentally made request-local fitting look correct.

BOTIE

Botie’s first test accidentally made request-local fitting look correct.

Choose an example

Current request batch

1

2

Stored training centre

Independent expected first transformed value

Absolute numerical tolerance

Centred readings and the anchored first-reading check

[
  [
    -2,
    2
  ],
  true
]

Follow the operations

1. Current request batch
1. 02
2. 16
2. Stored training centre
```
4
```
3. Subtract
1. 0-2
2. 12
4. Set a constant
```
0
```
5. Choose an entry
```
-2
```
6. Independent expected first transformed value
```
-2
```
7. Absolute numerical tolerance
```
0
```
8. Check a numerical contract
```
true
```
9. Collect two results
```
[
  [
    -2,
    2
  ],
  true
]
```
10. Result
```
[
  [
    -2,
    2
  ],
  true
]
```

This relationship belongs to a deterministic transform that processes examples independently. A batch-dependent model needs a different promise, and a passing relationship test still needs independently calculated result checks.

1. Begin with a claim the interface actually makes

Our centring function promises to subtract the saved training centre from each example independently. Adding a neighbour to the batch should therefore leave the first example’s result alone. A statement like that, intended to hold across accepted inputs, is called a property.

Check the promise before turning it into a test. A model that deliberately ranks an example against its neighbours should depend on the batch. Training-mode batch normalisation also uses batch statistics. The same independence test would be wrong for those operations.

A test that encodes the wrong promise can reject correct implementations.

2. Keep one calculated reference beside the relationship

The original batch happens to have the saved training mean. A helper that wrongly relearns the mean can therefore pass it. Move the same first reading into a different batch to separate the two implementations.

saved centre = 4
first batch = [2,6]
correct first result = 2 - 4 = -2

changed batch = [2,100]
correct first result = 2 - 4 = -2
wrong request mean = (2 + 100) / 2 = 51
wrong first result = 2 - 51 = -49

The unchanged first reading becomes a diagnostic probe of which reference the code used.

Observe the batch-independence relationship

This program uses a fixed fitted reference. It demonstrates the relation before the later challenge tests permutations of a named-feature interface.

Python

centre = 4
def transform(values):
    return [value-centre for value in values]

print(transform([2,6])[0])
print(transform([2,100])[0])

Output

-2
-2

The neighbour changes, but the first example’s transformation keeps its meaning.

3. Change representation while preserving the underlying request

Keep every feature name attached to its value, then rearrange the pairs. A predictor that reads by name should receive the same observation and produce the same result. Rearranging values without their names would describe another observation.

This is a metamorphic test: change an input in a specified way and check the relationship we expect between the two results. Equality is the relationship here. Other tests may expect a known change in the result instead.

The feature names must be distinct, and the prediction must be deterministic for the comparison to mean what we intend.

Check yourself

Questions on Parts 1 to 3

Question 1 of 2

Which first-output relationship should hold?

Use the information below.

A fitted row-wise transform subtracts stored centre 4. Requests[2,6] and[2,100] share the same first reading.

4. A property can pass for a useless implementation

A predictor that always returns zero is invariant to every feature permutation. It is still wrong for most requests. Combine relationship checks with independently calculated fixtures and other requirements.

Round-trip tests have a similar limitation: an encoder and decoder can share the same mistaken convention. A second source of expected meaning is useful when both sides could be wrong together.

Passing a finite collection of tests is evidence about those inputs and assumptions. It is not a proof that the whole model or interface is correct.

5. Choose comparisons appropriate to numerical output

Floating-point evaluation can change slightly when arithmetic is rearranged. Define a tolerance from the calculation and its consequences, rather than choosing a loose value merely to make tests pass. Check shapes before comparing arrays so broadcasting cannot conceal a missing axis.

A common array check uses an absolute tolerance plus a relative term. Near zero, the absolute term matters. Decide explicitly how nonfinite values should be treated; do not let two NaNs count as a successful measurement comparison by accident.

scalar absolute-error rule:
|actual - expected| ≤ absolute tolerance

common relative-plus-absolute rule:
|actual - expected| ≤ absolute tolerance + relative tolerance × |expected|

The workshop uses the first rule for one finite scalar. The exercise’s contract also uses a finite scalar prediction.

6. Generate cases that could contradict the claim

A property-based testing tool needs instructions for generating accepted inputs. Those instructions are called a strategy. For example, it could generate short lists of named finite measurements and then rearrange their pairs. Invalid-message tests ask a different question and need their own expected errors.

Our challenge keeps the list small and tries every permutation of its positions. That covers those particular reorderings completely. It still leaves other measurement values to be tested, and the number of reorderings grows too quickly to use this approach on a large feature list.

Factorial growth makes full permutation enumeration unsuitable for a large feature schema. Keep this exercise’s feature lists small.

7. Keep the counterexample after fixing the code

When a generated case fails, save the smallest useful input and the expected behaviour as an incident regression. That test should fail on the old implementation and pass on the repair. Preserve the incident’s contract and relevant model/configuration versions.

A smaller case helps diagnosis, but do not remove the condition that made the failure real. An extreme neighbour may be exactly what distinguishes fitted-state reuse from request-local fitting.

Record the seed or reproducible generated fixture when randomness participates. Avoid making a flaky retry the way a failing test disappears.

8. Test that the test can detect the known mistake

Run the property against a deliberately simple broken implementation in local test code: a positional feature reader, or a request-local mean. If the property never fails, inspect whether the generator changes anything relevant.

Keep that testing double separate from production code. It is a check of the test’s sensitivity to this known defect, not evidence that all other defects are covered.

The challenge includes both a correct name-aware predictor and a broken positional predictor. Neither calls a live model service.