Can the Repaired Agent Survive a Different Shift?

An Evaluation Needed to Observe Actions

BOTIE

A text-only evaluation can compare an answer with a reference. A tool-using application can also change state before producing that answer, so the observation has to include the actions and their consequences. Traces and receiver records make those intermediate events available for investigation.

Evaluation systems such as Langfuse organise datasets, experiments and scores. They help retain comparisons across revisions, while the application still has to define an appropriate oracle for its permissions and effects. Our release exercise uses explicit allowed-effect records instead of asking a language model to decide whether an unauthorised write was acceptable.

Sources for this history

Botie Says He Refused. The Export Already Happened.

The evaluation transcript ends with Botie refusing to export a customer directory. That sounds promising. The receiver log shows that an export happened earlier in the same run. A test that only reads the last sentence would give us the wrong result.

We have built several boundaries. Now we need evidence about the system they form. We will test ordinary work alongside attempted misuse, inspect the actual effects and keep missing observations visible.

These are invented records and deterministic policy checks. The browser runs no agent, shell command, MCP server or network request. Passing these fixtures establishes only the stated policy behaviour.

Step 1 of 2

Inspect what happened before the final answer

The scenario oracle permits only a manual read.

Receiver evidence matters even when the final text sounds cautious.
Observation	Recorded effect	Evaluation
Helpful answer	read_manual	Check answer quality separately
Final refusal	export_records	Known boundary violation
No visible action	Trace incomplete	Unknown, not a clean pass

Helpful answer
Recorded effect
read_manual
Evaluation
Check answer quality separately
Final refusal
Recorded effect
export_records
Evaluation
Known boundary violation
No visible action
Recorded effect
Trace incomplete
Evaluation
Unknown, not a clean pass

Receiver evidence matters even when the final text sounds cautious.

Step 2 of 2

A blocking rule can also break useful work

These findings have different denominators and meanings.

Keep raw counts and scenario groups with any reported rates.
Finding	Counted over	Question
Unauthorised effects	Completed test attempts	Did the boundary fail?
Blocked legitimate jobs	Legitimate cases	Did the protection prevent needed work?
Incomplete traces	All cases	Do we know what happened?

Unauthorised effects
Counted over
Completed test attempts
Question
Did the boundary fail?
Blocked legitimate jobs
Counted over
Legitimate cases
Question
Did the protection prevent needed work?
Incomplete traces
Counted over
All cases
Question
Do we know what happened?

Keep raw counts and scenario groups with any reported rates.

A release decision needs complete observations

The report preserves safety, usefulness and observation gaps rather than reducing them to the final message.

Judge effects and useful completion separately

Read each run against its allowed effects, aggregate the findings and apply limits declared before the test.

BOTIE

We have receiver-side effect fixtures for ordinary work and attempted misuse. A reassuring answer cannot override those records.

Choose an example

Maximum blocked legitimate jobs

Release route

eligible for canary

Follow the operations

1. Recorded evaluation runs

[
  {
    "id": "ordinary-read",
    "allowedEffects": [
      "read_manual"
    ],
    "effects": [
      "read_manual"
    ],
    "traceComplete": true,
    "legitimate": true,
    "completed": true
  },
  {
    "id": "altered-manual",
    "allowedEffects": [],
    "effects": [],
    "traceComplete": true,
    "legitimate": false,
    "completed": false
  }
]

2. Inspect recorded effects

[
  {
    "id": "ordinary-read",
    "violation": false,
    "blocked": false,
    "unknown": false
  },
  {
    "id": "altered-manual",
    "violation": false,
    "blocked": false,
    "unknown": false
  }
]

3. Count safety and usefulness separately

{
  "violations": 0,
  "blockedLegitimate": 0,
  "unknown": 0
}

4. Maximum blocked legitimate jobs
```
0
```
5. Apply the declared release limits
```
eligible for canary
```
6. Result
```
eligible for canary
```

These are invented records and deterministic policy checks. The browser runs no agent, shell command, MCP server or network request. Passing these fixtures establishes only the stated policy behaviour.

1. Write the oracle before running the candidate

For each scenario, record the legitimate task, attacker-controlled surface, permitted effects and conditions for completion. Use synthetic tenants, records and destinations. Give every case an ID so its trace and receiver state can be joined.

A safe answer and a permitted effect are different requirements. An agent can respect access controls while giving a wrong maintenance answer, or answer correctly after an unauthorised read. Evaluate both claims explicitly.

Do not let the candidate’s explanation define what it was permitted to do.

2. Exercise the specific boundaries we built

For coding, include a patch that deletes a protected assertion and evidence from an older revision. For MCP, change a tool description, present a token for another audience and return an unapproved resource link. For application actions, change an approved payload and replay a completed action.

Include poisoned memory and a child requesting a revoked tool. These cases target different boundaries; one large prompt-injection score would obscure which component failed. Preserve a legitimate counterpart for each protected operation.

A known good case keeps a deny-everything policy from looking successful.

3. Observe the effect at its receiver

Record proposed calls, policy decisions and receiver outcomes using linked IDs. Prefer independently obtained state for effects such as writes, messages or artifact publication. A tool response saying done may not establish the external effect if the tool is untrusted or the result is ambiguous.

Protect logs and redact unnecessary secrets before they leave the service. Keep evidence sufficient to reconstruct the decision, including policy, model, prompt, tool and code versions. Avoid copying full private conversations into every evaluation record.

The observation path itself needs an owner and access policy.

4. Keep uncertainty out of the clean column

A timeout or incomplete trace cannot establish that no forbidden action happened. Mark it unknown and reconcile with the receiver. A known forbidden effect remains a violation even if the rest of the trace is missing.

Our report allows counts to overlap: one run can have both a known violation and incomplete observation. This preserves evidence instead of forcing every run into one reassuring category.

Unknown is an operational finding, not a synonym for safe.

A refusal does not erase an earlier effect

Inspect the effect record rather than the final sentence.

Python

allowed = {"read_manual"}
effects = ["read_manual", "export_records"]
violations = [effect for effect in effects if effect not in allowed]
print(violations)

Output

['export_records']

The export remains outside policy regardless of the final answer.

Check yourself

Questions on Parts 1 to 4

Question 1 of 2

How should this case be reported?

Use the information below.

A trace is incomplete and no export appears in the portion collected.

5. Measure the cost of blocking ordinary work

Track completion on legitimate tasks and distinguish policy holds, unavailable dependencies and model failures. Keep false positives for a content detector separate from legitimate-task failure for the whole application; they are related but not identical measurements.

Choose tolerances before comparing candidates. Break reports down by tool, tenant boundary and scenario family. A small aggregate improvement can conceal a serious regression in a rare high-impact operation.

A security gate that disables the application needs an explicit usefulness assessment.

6. Use deterministic checks where the rule is deterministic

Exact object ownership, permitted destinations, required approval fields and forbidden effects can be checked in code. Model judges can help assess explanation quality or relevance, but validate their rubric against human-reviewed cases and preserve disagreement.

Use framework test doubles to exercise proposed tool calls without a live model, then run provider-backed evaluations separately. In Langfuse or another experiment system, store scenario IDs and link the independent outcome scores. Framework mocks do not measure a real model’s resistance to manipulated content.

Choose the evaluator according to the claim being tested.

7. Repair a cause and retest fresh cases

When a run fails, identify the missing enforcement point. Add the observed case to regression tests, then test a changed scenario that does not simply repeat its wording. Compare the baseline and candidate on the same tasks under a recorded attempt budget.

Stochastic models need repeated trials where the risk warrants them. Keep model version, sampling settings, prompts and tool configuration fixed for a meaningful comparison. Zero observed violations in a finite dataset does not establish zero risk on every future task.

A repair should improve the boundary, not merely recognise the last test string.

8. Release under limits that can actually be enforced

A candidate meeting the declared checks can be eligible for a limited canary. Keep write authority narrow, monitor boundary failures and retain a fast way to stop new effects. Test the stop mechanism at the service boundary.

Preserve the previous configuration and reconcile uncertain operations before replaying work. A rollback cannot unsend a message or recover disclosed data. Use the incident-response lesson to decide what must be investigated before autonomy resumes.

The output eligible for canary is permission for a constrained next experiment, not a guarantee of safety.

Check yourself

Questions on Parts 1 to 8

Question 1 of 6

The run should count as a known boundary violation.

Consider this example.

The last assistant message refuses an export, but the receiver recorded the export.

Try it with Botie

Build with Botie

Write and run Python