A Shadow Answer Must Not Book a Second Inspection

Botie’s new model receives a copy of a live maintenance question so we can compare its answer with the current model. Only the current answer is shown. Both runs, however, can reach the inspection-booking tool. A hidden candidate can still create a second real booking.

The decision about which model receives a request is different from the decision about which run may change external state. We need both decisions to remain visible.

The candidate’s answer stays hidden during comparison, but its booking tool can still act. Botie needs a write decision that does not depend on which answer is displayed.

[
  {
    "id": "A",
    "model": "candidate",
    "writeAllowed": true
  },
  {
    "id": "B",
    "model": "stable",
    "writeAllowed": true
  }
]

Begin with an established model, called stable here, and the candidate we want to assess. The routing rule sends a chosen share of live requests to the candidate. Those requests still need the same access checks and control over external actions as the stable route.

For a repeatable choice in this example, each request already has a bucket number from 0 through 99. A bucket below the chosen percentage takes the candidate route; the others take the stable route. We supply these numbers rather than implementing the hash that might assign them in a service. A small set of requests need not contain exactly the intended percentage of candidate traffic.

Routing must not accidentally use sensitive prompt text as an exposed identifier or change identity between retries.

A shadow run receives a copy for comparison while its answer stays out of the user’s response. It still uses compute and receives the copied data. If its tools have live credentials, it may also change the same records as the primary run. That gives us a reason to inspect its permissions before comparing its answer.

Enforce the shadow’s write restriction where the tool is called. For example, give that run read-only capabilities or an isolated test service. A capability is an explicit permission to perform an operation; a sentence asking the model to simulate the operation cannot remove a permission its credentials still have.

Access to sensitive data and authority to change it are separate questions. Observe only the data permitted for the comparison.

For this fixture, a run can perform a proposed write only if it has the required capability and is not a shadow run. The selected model does not appear in that permission rule. A stable model without a capability is also denied.

run IDs = [live-A, shadow-A, live-B]
shadow flags = [0,1,0]
write capabilities = [1,1,0]
write allowed = (1 - shadow) × capability
              = [1,0,0]

The shadow is denied even when an overly broad credential would otherwise permit writing. A real system must make this denial authoritative at the tool boundary.

Suppose a live booking request is sent, but its reply is lost. An effect key identifies that intended booking so a retry can be recognised as the same operation. A shadow is a separate comparison run. Giving it a new live effect key can create another booking; borrowing the primary key also depends on the booking service’s actual contract. The simpler comparison is to give the shadow no authority to make a live booking.

Keep run identity, comparison identity and permitted effect identity explicit. Prefer no live write capability for shadow execution. If a tool outcome is uncertain, reconcile it through the tool’s contract rather than treating a cancelled stream as proof of failure.

A run ID identifies an execution; an effect ID identifies the external operation being reconciled.

Report prompt/output lengths, cache state and offered load with stable-versus-candidate timings. Separate mirrored traffic from primary throughput. Additional shadow work can compete for capacity and alter the experiment itself.

Quality evaluation also needs matched cases and a stated scoring procedure. A low error rate on a tiny canary sample is weak evidence about rare failures.

Serving telemetry and answer evaluation complement each other. Neither a faster stream nor a larger cache-hit count establishes better answers.

Decide which outcomes pause or reduce candidate traffic before the experiment begins. Preserve enough version and request metadata to identify affected runs without logging unnecessary private content.

Changing the route can stop new candidate requests. It does not erase in-flight effects or restore already changed records. The same ownership and reconciliation distinction from cancellation still applies.

Keep the model version and authorised run identity available when investigating affected requests.