Prompting, Evaluation and Iteration
Give each failure somewhere useful to go
The right evidence arrives, but the answer is still wrong
Botie’s stale-manual repair helps one request, but another now exposes a bad decision. Build an inventory that lets the evidence change the diagnosis.
BOTIE
The inputs we are comparing
Reviewed task runs
The manual lookup is the common failure
[
{
"id": "a",
"passed": false,
"stage": "evidence"
},
{
"id": "b",
"passed": false,
"stage": "evidence"
},
{
"id": "c",
"passed": true,
"stage": null
}
]The repaired lookup exposes a decision failure
[
{
"id": "a",
"passed": true,
"stage": null
},
{
"id": "b",
"passed": false,
"stage": "decision"
},
{
"id": "d",
"passed": false,
"stage": null
}
]Result for the manual lookup is the common failure
{
"counts": {
"evidence": 2,
"decision": 0,
"execution": 0
},
"unresolved": []
}Every failed trace needs another observation
This check uses the pieces from “Separate the failures we can locate”.
BOTIE
- Reviewed task runs
[ { "id": "u2", "passed": false, "stage": null }, { "id": "ok", "passed": true, "stage": null }, { "id": "u1", "passed": false, "stage": null } ]
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
What evidence would justify moving an unresolved run into the decision bucket?
Treat a diagnosis as a claim supported by the trace, then test the proposed repair.
These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.
Work through the Python challenges →