The Answer Starts Late, Then Arrives in Bursts

Why Generating an Answer Changed the Serving Problem

BOTIE

Once a model has learned its parameters, using it to produce an output is called inference. A service has to arrange that work for the people sending requests. Putting several requests through a model together can use the hardware more efficiently. That arrangement is a batch. Generating text complicates it: the model produces a token, uses that growing text as context, and returns to produce another. One answer may finish while its neighbour still has a page to write.

In 2022, the Orca research serving system addressed batches that stayed fixed for an entire response. Its iteration-level scheduling reconsidered the participating requests after each generation step. That gave finished requests a way out and waiting requests a way in. We will return to that decision later. First, we need to see where a reader waits. The invented timestamps below explain timing measurements used by serving systems such as vLLM; they are not measurements of either implementation.

Sources for this history

Botie Has Read the Manual. Why Is the Screen Still Empty?

A technician asks Botie a short question, but the request also contains several pages from an equipment manual. The answer eventually appears quickly. Before that, the screen sits empty. Another request starts promptly, then produces words slowly.

Where is each request spending its time? The empty screen comes before the first visible piece of the answer. The later pauses occur while that answer is being continued. We will keep both in view, because improving one can leave the other unchanged.

Invented one-clock timestamps; no model executes and no GPU throughput is measured.

Step 1 of 1

One request on one clock

The timestamps belong to the same request and monotonic clock.

The empty-screen interval and the later pauses are different parts of the experience.
Event	Time in ms
Arrival	0
Dispatch	40
First token visible	140
Second token visible	170
Third token visible	230

Arrival
Time in ms
0
Dispatch
Time in ms
40
First token visible
Time in ms
140
Second token visible
Time in ms
170
Third token visible
Time in ms
230

The empty-screen interval and the later pauses are different parts of the experience.

Reading the supplied text is work too

The first interval and the later pauses remain visible separately.

Separate the first wait from later pauses

The technician sees a late start in one answer and a long pause in another. A single response duration does not show which wait changed.

BOTIE

The technician sees a late start in one answer and a long pause in another. A single response duration does not show which wait changed.

Choose an example

Request arrival in ms

Token emission timestamps in ms

1

2

3

First-token wait and later token gaps in milliseconds

Follow the operations

1. Token emission timestamps in ms
1. 0140
2. 1170
3. 2200
2. Set a constant
```
0
```
3. Choose an entry
```
140
```
4. Request arrival in ms
```
0
```
5. Subtract
```
140
```
6. Measure gaps between emitted tokens
1. 030
2. 130
7. Collect two results
```
[
  140,
  [
    30,
    30
  ]
]
```
8. Result
```
[
  140,
  [
    30,
    30
  ]
]
```

Invented one-clock timestamps; no model executes and no GPU throughput is measured.

1. Reading the supplied text is work too

Before continuing a prompt, the model processes the supplied tokens and calculates information that its attention layers will use again. Here that prompt includes the manual as well as the technician’s question. This initial processing is called prefill. Many input positions can be processed together, depending on the model and runtime, but the attached pages still require work before an answer can begin.

The request may also wait in a queue before the model starts. Converting text to tokens takes time, as can sending the response across the network and displaying it in the browser. Decide where the clock starts and stops before treating the whole empty-screen interval as model computation.

A server timestamp and a browser timestamp do not automatically share a clock. The teaching trace deliberately uses one clock.

2. Starting an answer and continuing it are different waits

Start a clock when the request reaches the boundary we have chosen. Stop it when the first token is emitted. That elapsed interval is time to first token. Afterward, the model continues the answer using the text accumulated so far. This continuation is called decoding. The timestamps let us distinguish starting the answer from keeping it moving.

The first token can be sampled from the prompt-processing result. Do not assume every runtime has an additional, separately timed “first decode” stage after prefill.

arrival = 0 ms
first visible token = 140 ms
time to first token = 140 - 0 = 140 ms

The technician waited this long before seeing any of the answer. Faster later tokens do not erase that initial wait.

Check yourself

Questions on Parts 1 to 2

Question 1 of 2

Which interval describes the initial empty-screen wait?

Use the information below.

A request arrives at 0 ms and its first token appears at 140 ms. Later tokens arrive at 170 and 230 ms.

3. Keep the pauses instead of hiding them in an average

Subtract neighbouring emission timestamps to obtain the gaps between tokens. A single-token response has no such gap. Here one pause is longer than the other, which a mean alone would conceal.

emission times = [140, 170, 230] ms
between-token gaps = [170 - 140, 230 - 170]
                   = [30, 60] ms
mean gap = (30 + 60) / 2 = 45 ms
arrival-to-last-token time = 230 ms

The number of between-token intervals is one smaller than the number of emitted tokens. Streaming chunks can contain several tokens, so chunk timestamps require a different interpretation.

4. More tokens per second can coexist with slower answers

Batching can keep an accelerator busier and increase the total tokens completed across requests. Each individual request can still wait longer between its own turns. System throughput measures completed work across a time window; it is not the reciprocal of every user’s latency.

Compare workloads with similar prompt and output lengths. Report distributions and offered load, including rejected or cancelled requests, rather than presenting one fast completed example as capacity.

A token rate from one response and an aggregate service token rate answer different questions.

5. Decide what an experiment would distinguish

Suppose the queue wait grows while the model’s measured processing times stay similar. That points us toward the arriving workload and the decision about when to admit requests. If attaching a longer manual instead increases prompt-processing time, we have a different question: could some repeated prompt work be reused? Long prompt processing can also delay answers already being generated, which gives us a reason to examine the scheduling policy.

We would need timestamps around the relevant stages to test these explanations on a real service. Repeat the measurement with realistic request arrivals and keep network delays visible. Some optimisations propose several continuation tokens before checking which can be accepted, a method called speculative decoding. Such methods change the relationship between model operations and emitted tokens, so the measured boundaries still matter.

The timestamps give us a specific question to investigate with stage instrumentation.

Check yourself

Questions on Parts 1 to 5

Question 1 of 4

Which list preserves both between-token gaps?