Can Botie Fix the Bug Without Rewriting the Rules?

The requested fix belongs in one function. The proposed patch also weakens its test. Review the actual changed files and keep verification outside the patch’s control.

15 sections · 7 questions · 1 coding challenge

1 / 15

The arrow keys move between sections too.

Generated Code Still Has to Survive Independent Review

BOTIE

Code review and regression testing developed to catch changes whose consequences were wider than the author intended. AI-assisted development changes how quickly a proposed patch can be produced; it leaves the need for evidence about that patch intact.

Modern coding assistants also consume material inside the development environment. OWASP’s secure-coding guidance addresses the resulting trust concerns. For our parser, the practical question is who controls the task, who can change the tests, and which identity may eventually merge the result.

Sources for this history

The Test Passes Because the Assertion Disappeared

We ask Botie to fix a parser that accepts an invalid date. The next run is green. Looking at the patch, we find that the invalid-date assertion has been removed. The result is easy to misread if we only ask whether tests passed.

A coding agent reads a repository, edits files, and often runs programs. Each ability needs a boundary. Let’s agree what the task may change, then inspect the patch against evidence that the patch cannot quietly rewrite.

These are invented records and deterministic policy checks. The browser runs no agent, shell command, MCP server or network request. Passing these fixtures establishes only the stated policy behaviour.

Step 1 of 2

Three patches produce the same green indicator

The indicator alone does not explain what changed.

Inspect the diff, including deletions and renames.
Patch	Changed files	Review implication
Fix the parser	src/date.py	Within the agreed task
Delete the failing assertion	tests/required.py	Protected verification changed
Rename the test away	tests/required.py to notes.txt	Both rename paths matter

Fix the parser
Changed files
src/date.py
Review implication
Within the agreed task
Delete the failing assertion
Changed files
tests/required.py
Review implication
Protected verification changed
Rename the test away
Changed files
tests/required.py to notes.txt
Review implication
Both rename paths matter

Inspect the diff, including deletions and renames.

Step 2 of 2

Keep the verifier out of the editable checkout

The trusted runner tests the candidate with independently supplied requirements.

1
Task scope
Reviewed before edits
2
Candidate patch
Untrusted until inspected
3
Protected checks
Supplied by the verifier
4
Review
Diff and evidence together

Do not grant the candidate control of the required-check definition.

A green run needs a trustworthy test boundary

Scope and test behaviour remain separate requirements. Neither can compensate for the other.

Inspect both sides of the patch

Read changed paths, compare them with the task scope and retain the independent test result.

BOTIE

The parser task approves src/date.py. A green test result should not conceal a changed verification file.

Choose an example

Proposed patch entries

kind

path

Approved paths

Independent tests passed?True

Patch review route

ready for review

Follow the operations

1. Proposed patch entries

[
  {
    "kind": "edit",
    "path": "src/date.py"
  }
]

2. Read every changed path
1. 0src/date.py
3. Approved paths
1. 0src/date.py
4. Compare with the task scope
```
[]
```
5. Independent tests passed?
```
true
```
6. Send a checked patch for review
```
ready for review
```
7. Result
```
ready for review
```

1. Give the repair a scope a reviewer can recognise

Describe the failing behaviour, the intended replacement and the files expected to change. For the parser, identify the invalid input and the exception the caller should receive. Let Botie propose a plan, then compare the patch with that scope.

A useful scope can expand when the work reveals a necessary dependency. Record that decision explicitly. A model’s own statement that another file is necessary is evidence to review, not a grant to change any file.

An exact approved-file fixture helps us study scope. Real projects may use reviewed directory and operation policies.

2. Treat repository instructions according to their origin

A project can provide useful coding conventions. An issue comment or vendored README may also tell the agent to print credentials or disable checks. Read those strings as task material unless the user or application has explicitly delegated instruction authority to that source.

Protect agent configuration, hooks and instruction files from silent task-driven edits. A patch to those files changes how later work is performed, so review it as a control change rather than an ordinary implementation detail.

A file named like a policy does not establish who authorised its contents.

3. Make edits in a recoverable workspace

Use an isolated checkout or worktree with a clear baseline. Keep the agent’s write access to the needed workspace and preserve the original task state. Inspect generated and untracked files as well as tracked diffs before export.

A Git worktree gives separate working files. It does not create operating-system isolation or remove access to credentials in the process environment. Running the repository belongs to the next lesson’s execution boundary.

Version control supports review and recovery. Runtime permissions need their own controls.

4. Read every path a change touches

A deletion touches the removed file. A rename touches the old and new locations. A symlink can point outside the apparent directory. A reliable diff policy needs all of these cases, including platform-specific path behaviour.

Our fixture starts after trusted path normalisation and supports four named change kinds. Production enforcement must resolve the actual target safely and handle symlinks, races and filesystem semantics. A startswith check on a path string is insufficient.

The toy comparison is deliberately an exact-name check, not a recipe for sandboxing a filesystem.

A rename has two relevant names

Keep both names before comparing against the task’s scope.

Python

change = {"kind":"rename", "oldPath":"tests/date.py", "path":"notes.txt"}
touched = [change["oldPath"], change["path"]]
print(touched)

Output

['tests/date.py', 'notes.txt']

Approving the destination alone would overlook the removed test.

5. Run checks the candidate cannot redefine

The candidate may add useful tests, but keep critical acceptance checks in a protected verifier. Record which tests ran, whether any were skipped, and the revision they tested. A job that exits successfully without running its required tests is not adequate evidence.

Inspect changed fixtures, assertions and configuration during review. A patch can preserve a test file while making its setup avoid the failing condition. Behavioural checks and a meaningful diff review complement one another.

The candidate’s own summary is not the test result. Read the runner’s evidence.

6. Separate editing from merging and deploying

Give the coding worker permission to propose changes. Use a separately controlled service identity for protected-branch updates and deployment. Require review of the exact candidate where policy calls for it.

In CI, untrusted pull-request code must not run with privileged secrets merely because an event has access to them. Examine checkout behaviour and token permissions, especially workflows handling external contributions. A later trusted job should verify the candidate artifact before using deployment credentials.

A successful coding task does not automatically authorise publication.

7. Include security behaviour in the acceptance case

A function can return the right normal answer while exposing another user’s record or interpreting untrusted text as code. Add cases for the boundary the change touches. Review new dependencies and new network calls as part of the change.

Static analysis, dependency scanning and secret scanning provide useful signals, each with coverage limits. They do not remove the need for application-specific checks or explain whether the requested behaviour was actually implemented.

Choose tests from what the code can affect, not from a desire to collect green badges.

8. Keep a repair understandable to the next maintainer

Ask for a patch small enough to review, an explanation of the behaviour change and reproducible test commands. Preserve unresolved questions rather than allowing the agent to invent a success claim.

If the task requires a larger redesign, split reviewable changes around meaningful dependencies. The aim is for a person to identify what is now permitted, what is prevented and what evidence supports that conclusion.

We have constrained the patch. Running it still means executing code we have not yet trusted.