← All workshopsRead What Can We Test Without Calling a Model?

Prompting, Evaluation and Iteration

Find the call that reached the wrong pump

The right tool reads the wrong asset

Concurrent requests finish in a different order. Botie needs to know whether each requested asset reached the correct tool.

BOTIE

Case B called read for P9, but it expected P8. Which result should the report preserve?
The inputs we are comparing

Expected single calls

Same in both cases

[
  {
    "id": "a",
    "tool": "read",
    "asset": "P7"
  },
  {
    "id": "b",
    "tool": "read",
    "asset": "P8"
  }
]

Observed single calls

Both calls match

[
  {
    "id": "b",
    "tool": "read",
    "asset": "P8"
  },
  {
    "id": "a",
    "tool": "read",
    "asset": "P7"
  }
]

The second call uses the wrong asset

[
  {
    "id": "b",
    "tool": "read",
    "asset": "P9"
  },
  {
    "id": "a",
    "tool": "read",
    "asset": "P7"
  }
]
Result for both calls match
{
  "passed": 2,
  "failed": 0,
  "missing": 0
}
Choose an explanation to test
The asset matches but the operation changes

This check uses the pieces from “Keep each contract attached to its case”.

BOTIE

Case X expects a read of P12. The recorded call creates a job for P12 instead. Case Y never returns. Predict the contract report.
Expected single calls
[
  {
    "id": "X",
    "tool": "read_manual",
    "asset": "P12"
  },
  {
    "id": "Y",
    "tool": "read_manual",
    "asset": "P13"
  }
]
Observed single calls
[
  {
    "id": "X",
    "tool": "create_job",
    "asset": "P12"
  }
]

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Which remaining question would require the intended live model rather than this recorded-call fixture?

A deterministic contract test checks application handling. Task evaluation checks the model’s choices within that application.

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

Work through the Python challenges →
Continue the courseWho Checks the Model That Checks Botie? →
← What Would Count as Doing the Job?