Did Botie Create the Inspection, or Did the Reply Get Lost?

The technician approves an inspection of P7. Botie submits the request, but the connection closes before the reply arrives. He tries again. The service now shows two inspection jobs for the same request.

From Botie’s side, the first attempt failed to produce a reply. From the service’s side, it may have succeeded. We need to keep those facts separate. This lesson builds a recovery decision that can reuse a recorded result, wait for a pending operation, or stop when another attempt would be unsafe.

Keep the receipt lookup, retry safety and run deadline visible on separate branches. Then combine them into one recovery action.

{
  "route": "retry same operation key",
  "delay_ms": 100,
  "result": null
}

A permission denial or invalid asset identifier will not improve because the same request is sent again. Route those permanent failures to repair or review. A temporary overload or interrupted connection may justify a retry, but the tool’s contract must establish whether repeating the operation is safe.

Keep unknown outcomes distinct from known rejections. For a write, a timeout can mean the service completed the effect without delivering a reply. For a read, another request usually avoids duplicate business effects, though it still consumes resources and can return changed data. Our fixture labels outcomes success, transient, permanent or unknown; real adapters map their documented error codes into those categories.

The model does not get to reinterpret a permission denial as an invitation to try different arguments until something works.

Create an operation key when the application accepts the approved intent. Persist it before sending the write. Reuse it when delivering that same operation again. A genuinely new inspection needs a new key, even if its asset and other fields match an earlier inspection.

The receiver binds the key to the caller’s scope and the relevant payload. The same key with a different pump or schedule must be a conflict. Avoid relying on the model to invent or remember this identity. It belongs to application state alongside the approved action revision.

A key in a request header is only useful if the receiving service enforces the contract.

A completed receipt can return the earlier work-order ID. A pending receipt tells us the operation is already under way. A status query or a bounded wait is appropriate; launching a second operation is not.

This requires an authoritative ledger. A stale cache or an unavailable status endpoint cannot prove that the operation never happened. When the receiving service offers neither an idempotency contract nor a reliable reconciliation path, retain the ambiguous outcome for review. Do not turn it into an automatic new write. The workshop’s absent receipt assumes a trustworthy snapshot within the stated service contract.

Approval answers whether we may perform the action. A receipt answers what happened when we tried. Neither replaces the other.

The service needs a unique operation identity, a fingerprint or canonical representation of its payload, a processing state, and a stored result. Concurrent deliveries of the same operation must compete at an atomic boundary so both cannot decide they are the first.

The exact transaction depends on where the effect happens. A database write and its receipt may share a transaction. An external email or another service cannot automatically join that transaction; outbox delivery and downstream idempotency may be needed. Receipt retention also matters. Once a receiver forgets a key, a late replay may be treated as new work. Read the actual API contract and keep retries inside its scope and retention window.

Our Python dictionary lookup demonstrates the decision only. It is not a concurrency-safe store.

receipts = {
    "R4-inspect-1": {"asset":"P4", "status":"completed", "result":"WO-18"}
}
key = "R4-inspect-1"
receipt = receipts.get(key)
if receipt is not None and receipt["status"] == "completed":
    print(receipt["result"])
else:
    print("No completed receipt in this snapshot")

WO-18

A per-attempt timeout bounds how long the caller waits for one operation. A run deadline bounds the whole recovery process, including delays and repeated attempts. Starting a fresh timeout after each failure can otherwise keep the run alive far longer than intended.

Count attempts that have actually started, including the first. Before scheduling another, check the remaining attempt allowance and whether its delay plus timeout fits the remaining deadline. Use a monotonic clock for elapsed-time accounting in a real process. Pass the remaining deadline to downstream calls where supported. Timing out locally does not roll back an effect already accepted elsewhere.

The supplied fixture clock is an integer millisecond counter so every branch can be tested without sleeping.

Repeated immediate requests can worsen an overloaded service. Exponential backoff increases the upper delay after successive attempts, and a cap prevents unbounded waiting. Jitter chooses a delay within a range so many clients do not all retry at the same instant.

Our full-jitter demonstration supplies a fraction between zero and one rather than drawing randomness during a test. We multiply that fraction by the capped upper delay and round down to whole milliseconds. In production, draw the random value per attempt and honour the service’s documented Retry-After guidance. Keep a separate total deadline; a capped delay alone allows infinitely many retries.

upper_delay = min(cap_ms, base_ms * 2 ** max(attempts_started - 1, 0))
delay_ms = floor(upper_delay * jitter_fraction)
next_attempt_fits = now_ms + delay_ms + timeout_ms <= deadline_ms

Equality is allowed in our scheduling fixture. Actual transport scheduling and cancellation are not precise guarantees that a remote operation finishes at that instant.

A provider SDK may already retry a request. A framework may retry a model response that fails validation. A tool adapter may retry its service call. If each surrounding layer also repeats the whole operation, the actual number of requests can grow much faster than the visible loop count.

Inspect the layers and give each failure one deliberate recovery owner. Count transport attempts separately from model turns and tool proposals. Record the operation key, attempt number, error category and final outcome without leaking sensitive request bodies. A validation retry asks for corrected content; a transport retry resends delivery. They have different purposes and may consume different budgets.

An agent’s friendly message that it is trying again should correspond to a recorded recovery decision in the application.

If the user cancels after submission, stop scheduling new work and request cancellation of any supported in-flight operation. The receiver may already have committed the effect. Preserve its operation identity and reconcile the final state before reporting that nothing happened.

For a workflow with several external effects, recovery may involve a compensating action such as cancelling a created appointment. That compensation is a separate authorised operation with its own possible failure. It does not erase the fact that the original effect occurred. Make partial outcomes visible to the person responsible for the request.

The next lesson will keep budget reservations for unresolved work rather than pretending a cancelled caller immediately receives its resources back.

Begin with the missing-reply case whose service supports a stable key. The recovery rule can permit another delivery within the deadline. Then load the completed receipt. The same construction should return its stored work-order ID immediately.

Change the payload under that key and check that the result becomes a conflict. Finally, remove the idempotency guarantee and leave the result unknown. Time remaining is no longer enough to justify another write. These cases test different reasons for stopping, so a single retry-until-success loop cannot satisfy them.

The result of this lesson is a recovery decision. Executing that decision still requires the live service contract and current authorisation checks.