Faster Batches Can Still Make a User Wait

Batching Trades Waiting for Shared Computation

BOTIE

Training software often processes several examples together because a processor can share work across them. An online service receives requests one at a time, and the first person in a group must wait while the others arrive. The batch that makes good use of the hardware can still give that person a slow answer.

The Clipper system, presented in 2017, treated prediction serving as a separate problem from training. Its adaptive batching adjusted how many queued requests a model processed together while respecting a latency objective. The useful change was to make batch size depend on the model's observed performance rather than assuming a larger batch was always better. Current servers such as Triton also expose controls for queue delay and batching.

Our first example follows requests already assigned to one batch. We will distinguish time waiting in the queue from time spent computing. A later scheduler has to make its decision using both, because sharing computation cannot recover time a request has already spent waiting.

Sources for this history

The GPU Got Faster. The First User Did Not.

Botie’s image model processes several images together more efficiently than one at a time. We wait for a larger batch and throughput improves. The first request, however, spends that waiting time in a queue.

We need to measure the experience of each request, not divide the batch runtime and call that the user’s latency.

A finite timing fixture omits network time and contention; it is not a throughput benchmark.

Step 1 of 1

The earliest request waits longest

All requests finish when the same batch completes.

Completion minus arrival measures each request’s latency.
Request	Arrival ms	Completion ms
A	0	14
B	4	14
C	6	14

A
Arrival ms
0
Completion ms
14
B
Arrival ms
4
Completion ms
14
C
Arrival ms
6
Completion ms
14

Completion minus arrival measures each request’s latency.

A batch shares computation, not arrival time

Each request keeps its own arrival when the batch shares a completion.

Follow the requests through one batch

A longer batching delay can hide behind higher execution throughput.

BOTIE

A longer batching delay can hide behind higher execution throughput.

Choose an example

Arrival times in ms

1

2

Dispatch time in ms

Execution duration in ms

Latency of each request in milliseconds

08
18

Follow the operations

1. Dispatch time in ms
```
0
```
2. Execution duration in ms
```
8
```
3. Add
```
8
```
4. Arrival times in ms
1. 00
2. 10
5. Subtract
1. 08
2. 18
6. Result
1. 08
2. 18

A finite timing fixture omits network time and contention; it is not a throughput benchmark.

1. A batch shares computation, not arrival time

Two independent image requests can often be calculated together because neither needs to remember the other’s earlier work. Such requests are stateless. A continuing conversation is stateful: its saved history must remain attached to that conversation. Combining those sequences needs additional scheduling rules as well as compatible input shapes.

Track arrival, dispatch and completion separately. Queue delay occurs before dispatch; execution time occurs after it.

Padding incompatible lengths can consume memory and computation. A larger batch is not automatically a better batch.

2. Calculate latency from the common completion

For the fixture, execution takes eight milliseconds and dispatch happens at six. Each arrival has a different wait.

completion = dispatch + execution = 6 + 8 = 14
arrivals = [0,4,6]
latencies = [14,10,8] ms
batch throughput = 3 / 0.008 = 375 requests/second

That throughput describes execution of this batch. It does not include idle time, queueing or transport overhead.

Separate queue delay from execution

A list retains one waiting time for each request.

Python

arrivals = [0,4,6]
dispatch = 6
print([dispatch-t for t in arrivals])

Output

[6, 2, 0]

All requests still have to execute after these waits. The earliest request has waited longest.

Check yourself

Whose waiting time are we measuring?

Question 1 of 2

What latency does that request see?

Use the information below.

Arrival 0, dispatch 6, execution 8 ms.

3. Set a delay limit before chasing utilisation

At low traffic, waiting for a full batch may take too long. A maximum delay lets a partial batch leave. Under heavy traffic, queues can grow even when batches dispatch immediately.

Compare latency distributions at realistic arrival rates. Include slow shapes and cold starts; one average runtime is insufficient.

A service objective should state whether it measures model execution or the full user-visible request.

4. Keep the measurement honest

Use monotonic clocks for elapsed durations within a process. Cross-machine timestamps need a clock model. Include cancellations and failures in the report rather than counting only successful fast requests.

The coding challenge checks impossible dispatch times so a negative wait cannot quietly improve the result.

The workshop is a timing model of one batch, not a measured GPU benchmark.