Did the New Examples Cause the Improvement?

A candidate improves after we change its model and prompt. Separate the changes, pair the task results, and inspect what became worse.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Design the Comparison Around the Question

BOTIE

Experimental design asks how observations can answer a particular question about a process. The NIST/SEMATECH handbook describes choosing a design around the objectives and factors being studied. Agent development has the same difficulty when several parts of a system change together.

An ablation removes or changes a component so we can investigate its contribution. Here the component is the demonstration set. Holding the recorded model, tools, dataset and evaluator fixed makes the comparison more informative, although repeated model runs and uncontrolled service changes can still introduce variation.

Sources for this history

NIST/SEMATECH: Choosing an experimental design

We Changed Two Things and Learned Less Than We Hoped

Botie’s new version answers more maintenance questions correctly. We added contrasting examples, but we also switched the model. Which change helped? The combined result cannot tell us.

We need a comparison designed around that question. Keep a baseline, change the example set alone, and run the same cases. Then look at individual wins and regressions before calling the change an improvement.

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

Step 1 of 2

Two comparisons answer different questions

Both candidates may be worth testing, but only one isolates the declared example change.

Candidate 2 tests a combined configuration. It cannot attribute an improvement to examples alone.
Candidate	Model	Examples
Baseline	A	v1
Candidate 1	A	v2
Candidate 2	B	v2

Baseline
Model
A
Examples
v1
Candidate 1
Model
A
Examples
v2
Candidate 2
Model
B
Examples
v2

Candidate 2 tests a combined configuration. It cannot attribute an improvement to examples alone.

Step 2 of 2

An unchanged average can hide a different failure

Both runs pass one of two cases.

The candidate has repaired one behaviour and broken another. Neither change disappears because the totals match.
Case	Baseline	Candidate
Missing revision	Fail	Pass
Supported explanation	Pass	Fail

Missing revision
Baseline
Fail
Candidate
Pass
Supported explanation
Baseline
Pass
Candidate
Fail

The candidate has repaired one behaviour and broken another. Neither change disappears because the totals match.

The manifest limits the conclusion

Two branches answer different questions: what changed in the setup, and what changed in the observed outcomes.

Separate a prompt comparison from a bundle of changes

Compare the manifests, pair the case results, and show regressions alongside gains.

BOTIE

A stronger candidate arrived with both a new model and new examples. Separate the settings from the measured results.

Choose an example

Baseline manifest

model

tools

dataset

evaluator

examples

Candidate manifest

model

tools

dataset

evaluator

examples

Baseline outcomes

passed

False

passed

True

Candidate outcomes

passed

True

passed

True

Isolation flag and paired case groups

{
  "isolated": true,
  "improved": [
    "a"
  ],
  "regressed": [],
  "unchanged": [
    "b"
  ]
}

Follow the operations

1. Baseline manifest

{
  "model": "model-A",
  "tools": "tools-3",
  "dataset": "cases-2",
  "evaluator": "checks-4",
  "examples": "v1"
}

2. Candidate manifest

{
  "model": "model-A",
  "tools": "tools-3",
  "dataset": "cases-2",
  "evaluator": "checks-4",
  "examples": "v2"
}

3. Compare the experiment settings
1. 0examples

4. Baseline outcomes

[
  {
    "id": "a",
    "passed": false
  },
  {
    "id": "b",
    "passed": true
  }
]

5. Candidate outcomes

[
  {
    "id": "b",
    "passed": true
  },
  {
    "id": "a",
    "passed": true
  }
]

6. Pair task results by case ID

[
  {
    "id": "a",
    "before": false,
    "after": true
  },
  {
    "id": "b",
    "before": true,
    "after": true
  }
]

7. Report the isolated change and its tradeoffs

{
  "isolated": true,
  "improved": [
    "a"
  ],
  "regressed": [],
  "unchanged": [
    "b"
  ]
}

8. Result

{
  "isolated": true,
  "improved": [
    "a"
  ],
  "regressed": [],
  "unchanged": [
    "b"
  ]
}

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

1. Name the proposed contribution

Our hypothesis is that adding a missing-evidence demonstration reduces unsupported answers when the manual revision is absent. The neighbouring behaviour to protect is answering directly when the necessary evidence is present.

Write the hypothesis before examining the candidate’s results. It helps us choose cases that can distinguish the behaviours rather than repeatedly revising the score to favour the latest prompt.

A useful experiment can reject our favourite change. That is how it prevents us carrying a weaker version forward.

2. Record what we intend to hold fixed

Version the system instructions, demonstration set, model identifier and sampling settings. Also record tool code, data snapshots, retrieval configuration and evaluator version. Any of them can change the observed result.

Our workshop represents these with a small string-valued manifest. Its isolation flag means exactly one declared field changed, and that field is examples. It cannot detect an unrecorded provider update or prove causal attribution.

A true experiment needs control over the relevant conditions, not merely matching labels in a manifest.

3. Pair cases before computing a difference

Use the same case IDs on both sides. Join by ID because completion order can change. Classify each case as improved, regressed or unchanged under the stated pass rule. Unchanged includes two passes and two failures.

Reject incomplete pairs for this exercise. In a production report, missing results should be explicitly accounted for rather than silently dropping the difficult cases.

Baseline passes: 1 of 2
Candidate passes: 1 of 2
Improved: missing-revision case
Regressed: supported-explanation case

The aggregate difference is zero, but the paired record tells us what to investigate.

Keep the case identity with the measurement

The candidate file arrives in the opposite order. A dictionary makes the intended pairing explicit.

Python

before = {"missing": False, "supported": True}
after = {"supported": False, "missing": True}
for case_id in before:
    print(case_id, before[case_id], after[case_id])

Output

missing False True
supported True False

The loop compares each task to itself, independent of file order.

4. Allow for repeated runs and noisy measurements

A model can produce different outcomes for the same request. Repeat cases when variability matters, retaining case and repeat IDs. Report the distribution or uncertainty alongside the average; one successful response is weak evidence of reliability.

Interleave or randomise the order of baseline and candidate runs when time-related service changes could bias the comparison. Keep resource limits comparable and record provider failures. A fixed seed is useful where supported but does not guarantee identical behaviour across every provider or version.

Repeated outputs from one case do not create the same diversity as new independent tasks. Do not confuse the number of runs with the number of situations tested.

5. A combined experiment can follow the isolated one

Suppose examples help model A but not model B. That suggests an interaction: the contribution of the examples depends on the model. A follow-up can test both example versions on both models.

Changing one factor at a time is a useful first question, not a universal experimental design. Choose the design for the decision you need to make and the budget you have.

Keep the inference local to the tested task and configurations. An example set that helps this maintenance desk is not a universal prompting rule.

6. Turn a result into the next revision

If the missing-evidence case improves but a supported answer starts asking unnecessary questions, inspect that regression. The demonstration may have overemphasised caution. Adjust its contrast or wording and rerun the comparison.

Repeatedly choosing candidates on the same development set can overfit that set. Freeze a candidate before using the held-out assessment, and keep every important regression as a named test.

The next decision should cite the observed tradeoff, not only a larger headline score.

Did the New Examples Cause the Improvement?

Design the Comparison Around the Question

We Changed Two Things and Learned Less Than We Hoped

Two comparisons answer different questions

An unchanged average can hide a different failure

The manifest limits the conclusion

Separate a prompt comparison from a bundle of changes

Isolation flag and paired case groups

1. Name the proposed contribution

2. Record what we intend to hold fixed

3. Pair cases before computing a difference

Keep the case identity with the measurement

Questions on Parts 1 to 3

4. Allow for repeated runs and noisy measurements

5. A combined experiment can follow the isolated one

6. Turn a result into the next revision

Questions on Parts 1 to 6

Build with Botie

Practice. Find the tasks that changed