Stop Accepting Work You Cannot Hold

A technician abandons a long answer and asks a different question. The website stops displaying the first stream, but the execution worker has not yet confirmed that it stopped. If Botie immediately promises that same memory to another request, both may still need it.

At the same time, more questions keep arriving. We need to track who still owns resources and decide how much unfinished waiting work the service will accept.

The technician has closed an answer, but its worker has not confirmed that the memory is free. More questions are arriving while that reservation remains held.

{
  "heldSlots": 2,
  "heldBlocks": 5,
  "queued": [
    "Q1",
    "Q2"
  ],
  "rejected": [
    "Q3"
  ]
}

The technician’s Stop action tells us the answer is no longer wanted. It does not yet tell us the worker has finished cleaning up. We mark that request cancel_requested and continue counting its reservation. Only the execution layer’s confirmed release lets us give the same storage to another request.

Our fixture calls that final state released. A real runtime may expose task completion, an abort acknowledgement or a different lifecycle event. Handle races with ordinary completion and repeated cancellation requests deliberately.

The state names are a teaching interface. The property we need is an authoritative end to resource ownership.

Requests that cannot start yet may wait in a queue. Accepting more of them still uses memory and can make every waiting person wait longer. Backpressure means passing that capacity problem back toward the sender: the service may ask it to slow down, or explicitly refuse work it cannot accept. Our model uses a limit on the number of waiting requests.

A queue-size limit bounds accepted waiting items in this model. It does not guarantee their waiting time, because service durations and arrivals can still vary.

waiting limit = 2 requests
already waiting = [Q1]
new arrivals = [Q2,Q3]
queue after admission = [Q1,Q2]
explicitly rejected arrivals = [Q3]

A refusal is observable work the service did not accept. Do not remove it from load reports and call the remainder the whole workload.

If an agent already submitted an inspection booking, ending text generation cannot unsend that booking. A timed-out reply might also conceal a committed tool result. Keep effect outcomes and inference-resource cleanup as different records.

A booking tool needs an identifier for the intended operation so that a retry can be recognised instead of making another booking. If the result is uncertain, the caller needs a way to check what happened. Those are the effect-handling problems from the earlier agent lessons. This workshop confines itself to resource ownership and the waiting queue.

A successful cancellation acknowledgement here says nothing about whether a previous tool write occurred.

A queued request can become useless before it reaches a worker. Propagate the original overall deadline, preserve response time, and avoid launching work when no useful allowance remains. An immediate retry loop after a capacity refusal can make overload worse.

An upstream producer needs a bounded retry policy appropriate to the operation, and a user may need a clear busy or cancelled outcome. The earlier deadline calculator supplies one component; it is not a complete queue scheduler.

The deadline answers when work stops being useful; the queue limit answers how much waiting work we accept.

Exercise cancellation before dispatch, during decoding and after ordinary completion. Check that cleanup happens once, that retained work remains counted, and that later requests do not inherit another request’s cache state.

Measure abandoned requests and confirmed releases separately. If they differ, the service may still be doing work no caller wants. The simulation cannot establish how promptly a deployed worker actually cleans up.

Inspect the release event beside its request ID so slow cleanup remains visible.