Keeping a Prediction Consistent Across Requests

The same input should have the same preparation and result whether it arrives alone or in a batch. Keep prediction free of accidental fitting and mutable shared state.

14 sections · 4 questions · 2 coding challenges

1 / 14

The arrow keys move between sections too.

Reducing Differences Between Training and Serving

BOTIE

Production experience collected in Rules of Machine Learning includes checking differences between training and serving features. A model can behave unexpectedly even when its parameter file is correct if the surrounding computation changes.

Our browser exercise does not deploy a server. It tests the core prediction procedure that an API could call, separating numerical correctness from network transport and operational concerns.

Sources for this history

Zinkevich: Rules of Machine Learning

Why Should a Prediction Change When Another Request Arrives?

Botie can run a model in a notebook. Serving it means other code supplies inputs, expects an output shape, and may call the predictor many times.

We will isolate a deterministic prediction function and check single-record and batch agreement. A network endpoint can call that function later; the mathematical contract should already be clear.

This numerical example begins with validated inputs and fixed parameters.

Step 1 of 3

Receive a batch

Rows are requests and columns are features.

Two records enter together. Column meanings must stay in the saved order.
Request row	Feature 1	Feature 2
1	1	2
2	3	4

1
Feature 1
1
Feature 2
2
2
Feature 1
3
Feature 2
4

Two records enter together. Column meanings must stay in the saved order.

Step 2 of 3

Apply one saved rule

Every row uses the same weights and intercept.

Weights: [2, −1]; bias: 1
Row 1: (1 × 2) + (2 × −1) + 1 = 1
Row 2: (3 × 2) + (4 × −1) + 1 = 3

Apply the same saved coefficients and bias to each incoming row, independently.

Step 3 of 3

Return one result per row

Preserve request order.

An invented example for this lesson.
Input row	Predictions
Request row 1	1
Request row 2	3

Request row 1
Predictions
1
Request row 2
Predictions
3

Request handling should preserve the fitted numerical procedure

Row order is preserved and all requests use the same parameter state.

Will a row get the same answer beside different neighbours?

The fitted rule is fixed across requests. Compare individual inputs with the same rows in a batch and check that row order and prediction values remain consistent.

BOTIE

Botie receives a validated feature table and fixed model parameters.

Choose an example

Feature rows

Saved weights

Bias

Predictions in request-row order

Follow the operations

1. Feature rows
1 2
3 4
2. Saved weights
1. 02
2. 1-1
3. Matrix times vector
1. 00
2. 12
4. Bias
```
1
```
5. Add
1. 01
2. 13
6. Result
1. 01
2. 13

Inputs must already satisfy the saved feature and preprocessing contract.

1. Separate Fitting, Prediction, and Transport

The prediction function should use the fitted state as it stands. It should not relearn a mean from the current request or overwrite a parameter. Otherwise one caller could change another caller's answer.

Keep fitting separate from this calculation. The transport layer can then parse the request, call the prediction function, and serialize its result. We can test the numerical procedure before adding the endpoint around it.

Our function receives only validated arrays and immutable model state.

2. Choose an Output Shape

For scalar regression, a batch of n records returns a vector of n predictions in the same order. A single record can be represented as a one-row matrix and still receive a one-element vector.

Avoid arbitrary squeezing of all singleton dimensions, which can make output type depend on batch size. Classification may need class order and separate scores, probabilities, and labels.

X shape = (n, feature_count)
w shape = (feature_count,)
Prediction shape = (n,)
prediction = X @ w + bias

Shape is part of the public interface, not an implementation accident.

3. Check Batch and Single-Record Agreement

Apply the same prediction function to a whole batch and to its rows individually. Concatenated individual outputs should agree with the batch outputs within a stated tolerance.

Use evaluation mode for models with training-dependent layers. Batch normalisation and dropout must follow their inference rules rather than learn from each request.

The example is deterministic linear inference, so the agreement should be straightforward.

Keep the prediction rule separate from the request

A validated request supplies two features. The already fitted coefficients determine how to combine them; serving the request does not fit new coefficients.

Python

import numpy as np
features = np.array([2., 4.])
saved_weights = np.array([1., 0.5])
saved_bias = 0.0
contributions = features * saved_weights
print(contributions.tolist())
print(float(contributions.sum() + saved_bias))

Output

[2.0, 2.0]
4.0

Both features contribute to one answer. Practice packages this rule for a batch and checks that the same row gets the same answer when served alone.

4. Distinguish Numerical Failure From Service Failure

A valid mathematical function can still sit behind a service that times out, overloads, or returns inconsistent versions. Measure latency, throughput, error rates, and resource use separately from model accuracy.

Preserve request-to-response ordering when batching. Include a model version with results so later investigations can identify which state produced them.

The workshop computes a batch prediction; the next lesson identifies the state responsible for that prediction.

5. A Batch Should Not Change the Meaning of One Record

A deterministic fitted predictor should return the same answer for a record alone or beside other records, unless the model explicitly uses batch context. Recomputing a batch mean during serving breaks that property.

Keep the fitted transform fixed, preserve output order and return one result or a documented error for each requested record. Test a short final batch and an empty batch according to the contract.

6. Check Single-Record and Batch Agreement

The same prediction function can serve both forms by representing one record as a one-row matrix. That keeps the feature axis in the same position and avoids special-case arithmetic.

The service wrapper can then handle request parsing and response formatting without changing the mathematical rule.

Use one numerical path for both request sizes

Two features are transformed with saved training means.

Python

import numpy as np
mean = np.array([10.,50.])
weight = np.array([2.,.1])
def predict(X):
    return (X-mean)@weight+3.
X = np.array([[12.,60.],[15.,40.]])
batch = predict(X)
single = np.array([predict(X[i:i+1])[0] for i in range(len(X))])
print(batch.tolist())
print(bool(np.array_equal(batch,single)))

Output

[8.0, 12.0]
True

This verifies batch consistency for the supplied rule. The surrounding service still needs to preserve those rows and their request identifiers.

7. Limit Work before Starting Expensive Computation

A valid shape can still request an unreasonable amount of memory or compute. Define supported batch sizes, input dimensions and time budgets. Measure end-to-end latency, including parsing and feature preparation, rather than only a matrix multiplication.

Batching can improve throughput while increasing waiting time for an individual request. Choose that tradeoff for the application. Avoid retaining unbounded request arrays while waiting for a larger batch.

8. A Timeout Is Not a Numerical Prediction

Keep service failures separate from model outputs. A timed-out request should not quietly return a zero prediction, which downstream code may treat as a valid estimate.

Include the model version with successful results and retain request identifiers for later outcome matching. Retries should follow a documented policy, especially if a prediction also triggers an external action. The example predictor is pure; a production action workflow needs its own duplicate-handling rules.