Serving Models and Agent Runs
A model that fits and an agent that answers once still have to handle concurrent requests. Measure what users wait for, carry deadlines through tool calls, and account for conversation memory. Then decide when a cached answer really belongs to the next request.
Latency under real workloads
A person waits from the moment they send a request, including any time it spends in a queue. An agent may make several tool calls during that same wait. We’ll measure the time that actually passes for the caller.
- 01Ready to exploreFaster Batches Can Still Make a User Wait
Compare throughput with request latency before choosing an inference batch policy.
- 02Ready to exploreA Tool Call Cannot Spend the Whole Deadline Again
Propagate remaining time through an agent’s tool calls and preserve an explicit response margin.
Memory and reuse
A saved conversation occupies memory. A saved answer is useful only if it still answers the current question. Work out what each kind of reuse requires.
- 03Ready to exploreThe Model Fits Until the Conversations Grow
Account for transformer key/value cache growth separately from model weights.
- 04Ready to exploreA Cached Agent Answer Belongs to a Particular Context
Separate response reuse from semantic similarity by preserving tenant, evidence and model identity.