Computer Systems and Infrastructure
Why can a model answer one person quickly, then slow down when several people ask at once? Follow a request from its arrival to its final answer and measure where the time goes.
Serving Models and Agent Runs
A model that fits and an agent that answers once still have to handle concurrent requests. Measure what users wait for, carry deadlines through tool calls, and account for conversation memory. Then decide when a cached answer really belongs to the next request.4 lessonsLLM Inference and Capacity
A language model answers one person quickly, but slows down when several people ask questions at once. Why? We’ll follow what it does with a prompt and what happens as it writes an answer. That gives us a reason to look at shared calculations and the memory each conversation occupies.5 lessons