LLM Inference and Capacity
A language model answers one person quickly, but slows down when several people ask questions at once. Why? We’ll follow what it does with a prompt and what happens as it writes an answer. That gives us a reason to look at shared calculations and the memory each conversation occupies.
When the answer arrives
The model has to process the prompt before it can emit its first token. It then produces more tokens over time. We’ll measure those waits separately so we know which part of the work we are trying to reduce.
- 01Ready to exploreThe Answer Starts Late, Then Arrives in Bursts
Botie’s answer can start late or pause halfway through. Follow those two waits separately before deciding what to change in the service.
- 02Ready to exploreThe Same Manual Can Be Read Once, Until Its Prefix Changes
Several questions begin with the same manual. Work out which earlier attention calculations can be reused when the question or model configuration changes.
Serving a changing workload
A request can finish while others are still generating answers. Work out when its memory becomes available for the next request, including what happens when somebody clicks Stop.
- 03Ready to exploreLet the Next Request In Without Overfilling the Cache
An answer finishes while another keeps generating. Give the next waiting request a place only when its saved attention state will fit too.
- 04Ready to exploreStop Accepting Work You Cannot Hold
The reader clicked Stop, but the worker may still hold the request’s memory. Track that release before promising the space to another request.
- 05Ready to exploreA Shadow Answer Must Not Book a Second Inspection
Botie’s candidate model can make a real booking even when its answer stays hidden. Compare model versions while keeping control of which runs may write.