The Model Fits Until the Conversations Grow

Reusing Attention State Consumes Memory

BOTIE

A language model generates a response one token at a time. Earlier tokens still matter to later ones. Saving the intermediate arrays called attention keys and values avoids calculating them again at every generation step, but the saved state grows as conversations grow. Fitting the model's weights in memory does not establish that all those conversations will fit too.

Kwon and colleagues studied this serving problem in their 2023 PagedAttention paper. Requests had different, changing cache sizes, so reserving large contiguous regions could leave memory unused inside or between allocations. Their design borrowed paging ideas from operating systems: manage attention state in smaller blocks and keep a record of which blocks hold each sequence. The vLLM serving system used that arrangement to support more useful work within its memory budget.

Before comparing ways to reserve memory, we need to count the values a conversation actually requires. This lesson calculates the storage for those arrays first. A real server also has bookkeeping overhead and temporary arrays, so its measured memory will exceed this deliberately isolated quantity.

Sources for this history

Kwon and colleagues: PagedAttention, 2023

We Loaded the Model. Why Did the Next Conversation Run Out of Memory?

Botie’s language model loads successfully onto the accelerator. Several long conversations later, the service runs out of memory. The parameter count did not change. The stored attention keys and values did.

In attention, each earlier token has a key used for comparisons and a value containing information a later position can gather. Saving those arrays avoids calculating them again for every new token. This is the key/value cache, often shortened to KV cache; the arrays are also called tensors. Its memory grows with the tokens retained for active sequences, so the weights alone cannot tell us how many conversations fit.

Dense logical storage only; no prefix sharing, compression, block overhead or temporary buffers.

Step 1 of 1

Count keys and values per cached token

A small teaching model uses these dimensions.

Keys and values together require 32 bytes per cached token in this fixture.
Dimension	Value
Layers	2
KV heads	1
Head dimension	4
Bytes per scalar	2

Layers
Value
2
KV heads
Value
1
Head dimension
Value
4
Bytes per scalar
Value
2

Keys and values together require 32 bytes per cached token in this fixture.

Weights and conversation state have different lifetimes

The shared weights do not remove per-sequence cached token storage.

Budget the state of active conversations

Botie’s model fits, but active token storage keeps growing.

BOTIE

Botie’s model fits, but active token storage keeps growing.

Choose an example

Retained tokens per sequence

1

2

Dense K and V bytes per token

Estimated KV bytes across active sequences

Follow the operations

1. Retained tokens per sequence
1. 03
2. 15
2. Add a list
```
8
```
3. Dense K and V bytes per token
```
32
```
4. Multiply
```
256
```
5. Result
```
256
```

Dense logical storage only; no prefix sharing, compression, block overhead or temporary buffers.

1. Weights and conversation state have different lifetimes

The model weights are shared across requests for one loaded version. Cached keys and values belong to token positions in active sequences. Generated tokens extend that state.

Use the number of key/value heads, which can differ from query heads in grouped-query attention. Count both key and value tensors.

Some servers share prefixes or use sliding windows. Those change the effective retained tokens and must be reflected in a more detailed estimate.

2. Multiply the dimensions, then count active tokens

For uncompressed dense cache entries with no sharing, multiply the storage per token by the sum of retained sequence lengths.

bytes per token = 2 × layers × KV heads × head dimension × bytes per scalar
example = 2 × 2 × 1 × 4 × 2 = 32
lengths = [3,5]
KV bytes = 32 × (3 + 5) = 256

The first factor of two counts keys and values. It is unrelated to the numerical precision.

Inspect one token’s dimensions

Keys and values share the same dimensions in this simplified cache model.

Python

layers, kv_heads, width, scalar_bytes = 2,1,4,2
print(2*layers*kv_heads*width*scalar_bytes)

Output

This scalar is bytes per retained token. It still needs the total retained token count to describe the active workload.

Check yourself

Which dimension grows during generation?

Question 1 of 2

How many bytes per token for keys and values?

Use the information below.

2 layers, 1 KV head, head dimension 4, 2 bytes per scalar.

3. Admission needs room for future tokens

A request can fit at admission and grow beyond the limit during generation. Bound generation length or reserve capacity for the configured maximum. Decide what happens when the bound is reached.

Physical allocation can round up to blocks. Fragmentation and temporary attention buffers make measured memory larger than the simple logical estimate.

Do not advertise the logical estimate as an exact GPU capacity. Compare against measured peaks under realistic concurrent lengths.

4. Tie memory policy to user-visible behaviour

A server may queue, reject or shorten requests under pressure. Each choice changes latency or output completeness. Record these outcomes rather than counting only successful requests.

Use this estimate with the earlier deadline model: admitting every request can leave all of them waiting too long.

The coding function deliberately exposes each dimension so a head-count or precision change is testable.

5. A request scheduler must leave headroom

A server may allocate cache in blocks and reserve space before every token is generated. Logical tensor bytes are therefore a lower-level accounting quantity, not a direct promise about admission capacity.

Measure peak allocated memory while varying concurrent lengths, batch size and output limits. Leave headroom for temporary buffers and exceptional paths. If a request must be rejected or queued, make that behaviour visible to the caller and the latency report.

Memory policy and deadline policy must be evaluated together. A queue that avoids an allocation failure can still miss every response deadline.