The Same Manual Can Be Read Once, Until Its Prefix Changes

Botie answers several questions about the same manual. Each request begins with the same instructions and document, followed by a different question. Reprocessing that unchanged beginning can repeat substantial work.

Could the next request use the calculations already made for that beginning? Sometimes. We need to know exactly which tokens stayed the same and whether the saved arrays were produced by the configuration now answering the question. A matching document name alone does not tell us either.

Botie has already processed this manual for another question. Which of those saved calculations still belong to the current request?

4

For repeated requests, we can retain the keys and values calculated for their shared leading tokens. That is prefix caching. The new question still has to be processed, and the model still has to generate its answer. This differs from keeping a completed answer and returning that same text later.

Reusing attention state can reduce repeated prompt processing. It does not establish that retrieved facts remain current or that the generated answer will be correct.

The earlier context-cache lesson concerned whole answers. Here the reused object is internal attention state.

A cached position was calculated with the earlier tokens in view. If a token near the front changes, the later context has changed too, even when some later IDs happen to match. Start comparing at the beginning and stop at the first difference.

A prompt template assembles messages and tool descriptions into the input the model receives. The tokenizer converts that input into IDs. Changing how either step works can alter the actual beginning, even when the technician has typed the same question.

cached IDs = [10,11,12,13,20,21]
incoming IDs = [10,11,12,13,30,31]
leading match = 4 tokens

early-edited IDs = [10,99,12,13,30,31]
leading match = 1 token

Treat these as illustrative IDs. A real tokenizer need not assign one ID to each displayed word.

Our teaching cache stores complete blocks. Round a matching prefix down before reuse because a partially matched block is not reusable under this rule. Capacity reservations go in the opposite direction: a sequence needing part of another block still needs that whole allocation.

Neither rounding direction is a statistical estimate. It follows the allocation unit.

tokens per block = 2
matching prefix = 3 tokens
reusable tokens = floor(3 / 2) × 2 = 2

reservation = 3 tokens
required blocks = ceil(3 / 2) = 2

The workshop returns reusable tokens, not a block count. Keep the unit visible at the output.

The same input IDs can produce different arrays after a model or adapter changes. An adapter is an additional learned modification used with the model. The array’s storage layout and numerical format matter as well. For this exercise, a six-field record describes the model, tokenizer, adapter, prompt template, tenant namespace and KV format. We require all six fields to agree before counting reusable tokens.

Actual runtime keys and supported models differ. vLLM documents prefix hashes and cache salts, including isolation between requests with different salts. Our contract is a teaching policy, not a statement that every server uses these exact six fields.

The tenant namespace says which group of requests may share this saved work under our application policy. The application still needs authentication and permission checks around the cache.

Each retained token has keys and values at several attention layers. Within a layer, several KV heads each store a vector of numbers. These dimensions determine the memory calculation below. Grouped-query attention lets several query heads use a smaller set of KV heads, so use the KV-head count when counting the stored arrays. Lower-precision storage and splitting arrays across devices change the accounting.

illustrative configuration:
20 layers, 8 KV heads, 64 coordinates per head
2 bytes per coordinate

bytes per token = 2 × 20 × 8 × 64 × 2
                = 40,960 bytes = 40 KiB

This excludes weights, temporary buffers, allocator metadata and duplicated or shared storage. It is an accounting example, not a model capacity recommendation.

Keep stable instructions before variable questions when that preserves the intended prompt. Measure the tokens actually reused and the first-token delay, rather than assuming every repeated word saves work. Eviction, block boundaries, cold requests and limited cache capacity can reduce reuse.

Changing a document version should trigger the appropriate compatibility or prefix change. Reusing an old answer or old retrieval result simply because token processing was cached is a different and potentially incorrect decision.

Keep enough version information to recompute an entry when its compatibility check fails.