What Should the Prototype Teach Us?

A prototype should answer a question that could change the design. Test interacting choices together instead of changing everything and guessing what helped.

14 sections · 4 questions · 1 coding challenge

1 / 14

The arrow keys move between sections too.

One Change at a Time Can Miss an Interaction

BOTIE

Experimental design gives engineers ways to learn from planned changes to a process. NIST’s handbook describes factorial designs, which include combinations of factor levels. Those combinations let us investigate whether the effect of one factor depends on another.

Our prototype varies batch size and parameter format. We will calculate a difference of differences in illustrative timing measurements. This is a descriptive contrast, not a complete statistical experiment or a fitted significance test.

Sources for this history

NIST/SEMATECH: Choosing an Experimental Design

Batching Helped Yesterday. Why Is It Slower Today?

Botie tries a larger batch and a smaller numerical format to make inference cheaper. The combined result looks promising. But how much came from each change, and would the smaller format help at the original batch size?

We can answer that with a small planned comparison. Run every combination of the two choices, keep the workload comparable, and inspect whether one change alters the effect of the other.

Timings are invented total batch durations, not per-request latency measurements.

Step 1 of 2

Run every combination

The rows are batch sizes and the columns are formats. All measurements use the same device and image shape.

The format change reduces total batch time by different amounts.
Batch	Original format	Smaller format
Small	40 ms	30 ms
Large	80 ms	40 ms

Small
Original format
40 ms
Smaller format
30 ms
Large
Original format
80 ms
Smaller format
40 ms

The format change reduces total batch time by different amounts.

Step 2 of 2

Compare changes, not just final numbers

Subtract original-format time from smaller-format time within each row.

The interaction contrast compares these effects. Negative means a larger time reduction at the large batch.
Batch	Format effect
Small	−10 ms
Large	−40 ms

Small
Format effect
−10 ms
Large
Format effect
−40 ms

The interaction contrast compares these effects. Negative means a larger time reduction at the large batch.

Use a prototype to separate competing explanations

Every subtraction holds one factor fixed before comparing the effects across the other factor.

Does the format change help equally at both batch sizes?

Enter the four batch-duration measurements in the stated order. Compare the format change at the large batch with its effect at the small batch.

BOTIE

Botie measured all combinations of batch size and parameter format. He needs to see whether the format effect depends on batch size.

Choose an example

Batch durations: small/original, small/smaller, large/original, large/smaller (ms)

Difference of format effects (milliseconds)

-30

Follow the operations

1. Batch durations: small/original, small/smaller, large/original, large/smaller (ms)
1. 040
2. 130
3. 280
4. 340
2. Set a constant
```
3
```
3. Choose an entry
```
40
```
4. Set a constant
```
2
```
5. Choose an entry
```
80
```
6. Subtract
```
-40
```
7. Set a constant
```
1
```
8. Choose an entry
```
30
```
9. Set a constant
```
0
```
10. Choose an entry
```
40
```
11. Subtract
```
-10
```
12. Subtract
```
-30
```
13. Result
```
-30
```

These are descriptive contrasts from invented timings. Repetition, measurement controls, and task-quality checks are still needed.

1. Ask a question whose answer would change the plan

“Try optimisation” is too vague to judge a prototype. A useful question here is whether the smaller format reduces batch duration enough to justify its conversion work, and whether that depends on batch size.

State what would count as useful evidence before running the trial. If prediction quality falls below a required limit, a faster run will not rescue that candidate.

A prototype can reject an idea cheaply. It need not become the final service to be valuable.

2. Name the factors and the response

A factor is something we deliberately vary. Our factors are batch size and parameter format, each with two chosen levels. The response is total time to process a batch. Keep the device, runtime version, and image dimensions controlled or recorded.

A large batch contains more images, so total duration alone does not establish per-image throughput or interactive latency. Calculate the relevant additional metric before choosing a deployment setting.

The four-cell table keeps each timing attached to the choices that produced it.

3. Calculate each format effect at a fixed batch size

Within the small batch, subtract original-format duration from smaller-format duration. Repeat for the large batch. Holding batch size fixed within each subtraction avoids attributing the extra work of a larger batch to format alone.

Small-batch format effect = 30 − 40 = −10 ms
Large-batch format effect = 40 − 80 = −40 ms

A negative effect means a shorter batch duration. The unequal changes suggest that the choices interact in these measurements.

4. Subtract the two effects

Our interaction contrast is the large-batch format effect minus the small-batch format effect. A zero contrast means equal absolute format effects at these selected batch sizes. It does not establish equal percentage changes.

Different statistical models scale interaction coefficients differently. We report this raw difference of differences, with its units and sign convention, so there is no hidden factor in the result.

Interaction contrast = (40 − 80) − (30 − 40) = −30 ms

The format reduction is associated with a larger absolute time saving at the large batch in this table.

Read a factorial table by row

Rows represent small then large batches; columns represent original then smaller formats.

Python

import numpy as np
times = np.array([[20., 18.], [50., 40.]])
format_effect = times[:, 1] - times[:, 0]
print(format_effect.tolist())

Output

[-2.0, -10.0]

The colon selects both rows. Column subtraction compares formats at the same batch size. The challenge adds the interaction contrast.

5. Repeat and vary the execution order

A single timing can reflect background work, warming, or thermal behaviour. Repeat each condition and vary the order so a gradual slowdown does not always affect the same candidate. Keep warm-up and measurement procedures consistent.

When operating conditions differ in a known way, compare within blocks such as devices or sessions. Preserve the individual measurements so we can inspect spread rather than only a rounded mean.

Repeated measurements support uncertainty estimates. Four isolated numbers cannot by themselves establish statistical significance.

6. Measure whether the prediction still does its job

Numerical changes can alter outputs. Evaluate the converted model on the same representative labelled cases and inspect important disagreements. Changing batching should also preserve the intended relationship between each input and output.

Keep preprocessing and evaluation mode consistent. A timing benchmark that accidentally enables a training-dependent layer may compare different behaviours as well as different speeds.

The prototype should return both operational evidence and task-quality evidence. They answer different design questions.

7. Decide what to try next from the pattern

If the format change helps only at larger batches, investigate the runtime’s execution choices and whether real traffic can fill those batches in time. A service with sparse arrivals may spend its savings waiting to form a batch.

A physical inspection station might instead have a predictable frame stream. The same benchmark can lead to different designs because the work arrives differently.

Botie’s workshop calculates the contrast. The conclusion remains conditional on this workload, these factor levels, and the measurement procedure.