Prompting, Evaluation and Iteration
Make the prompt experiment answer one question
BOTIE
Examples, experiments and workspace controls
Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.
Try your own inputs
Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.
Context for this challenge
Pair baseline and candidate records by ID, preserving baseline order despite shuffled candidate results.
A stronger candidate arrived with both a new model and new examples. Separate the settings from the measured results.
Inspect the component calculation
Pair task results by case ID( Baseline outcomes: Baseline outcomes Candidate outcomes: Candidate outcomes )
Examples and results
- Baseline outcomes
[ { "id": "a", "passed": false }, { "id": "b", "passed": true } ]- Candidate outcomes
[ { "id": "b", "passed": true }, { "id": "a", "passed": true } ]- Required result
[ { "id": "a", "before": false, "after": true }, { "id": "b", "before": true, "after": true } ]
- Baseline outcomes
[ { "id": "a", "passed": false }, { "id": "b", "passed": true } ]- Candidate outcomes
[ { "id": "b", "passed": false }, { "id": "a", "passed": true } ]- Required result
[ { "id": "a", "before": false, "after": true }, { "id": "b", "before": true, "after": false } ]
Try a fresh construction
This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.
Stuck? Inspect a worked construction
The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.
Assumptions and test precision
These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.
Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.
Nothing in the manifest changes
This check uses the pieces from “Pair the outcomes and inspect the settings”.
BOTIE
- Baseline manifest
{ "model": "M2", "examples": "e4" }- Candidate manifest
{ "model": "M2", "examples": "e4" }- Baseline outcomes
[ { "id": "x", "passed": false }, { "id": "y", "passed": false } ]- Candidate outcomes
[ { "id": "y", "passed": false }, { "id": "x", "passed": true } ]
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
How would you test whether the examples help model A but hurt model B?
When factors may interact, compare their combinations deliberately instead of attributing a bundled result to one change.
These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.
Work through the Python challenges →