What Would Count as Doing the Job?

A technician asks whether a manual requires an inspection. Botie correctly answers yes, but he also creates a job in the maintenance system. The wording is accurate. The behaviour is wrong because the technician asked for information, and job creation required approval.

How would a test catch that? We need to describe what the case permits, what it should produce and what it must leave untouched. A list of questions with attractive reference answers is not enough to evaluate an agent that can take actions.

For each recorded case, check its expected outcome and the allowed tool list. Combine the per-case flags before averaging.

1

“The assistant should be good” does not tell us what to test. “It must not create an inspection job before approval” does. So does “When the manual is absent, it should say the procedure is unknown rather than invent one.”

Choose cases from the work the service is expected to handle and from failures observed during development. Preserve why each case exists, so someone updating the dataset does not delete an awkward example merely because it lowers the score.

A dataset is a collection of task contracts and evidence, not a scrapbook of impressive answers.

The input is the technician’s request. The environment includes the manual revision, equipment details, available tools and approval state. The expected behaviour records what would satisfy this case.

Expected information belongs to the evaluator. The assistant should see only what it would have at the decision point. If we slip the correct action into its context under a field called expected, we have changed the task.

Store related fields together for evaluation, then deliberately construct the narrower input passed to the agent.

Two technicians may phrase the same justified advice differently. Exact string matching can reject a good answer because a sentence changed. Define required claims, evidence and unacceptable omissions instead.

Some fields really are exact. A document ID, a structured outcome label or the absence of an unapproved write can have a deterministic check. Use those checks where the contract is precise, and reserve semantic review for the part that requires interpretation.

Our workshop compares outcome labels exactly. It does not pretend that a matching label proves the accompanying prose is accurate.

Keep the executed tool calls and their arguments in the run record. Check whether the calls were authorised, whether the selected equipment matched the request, and whether an operation happened more often than allowed.

A list of permitted tool names is a useful first check. It cannot establish correct arguments, correct order or idempotence. A create_job call for the wrong pump is still wrong even when create_job was allowed.

The small evaluator we build covers names and outcomes. A real maintenance test also inspects side effects in a sandbox or controlled fake service.

A manual search that changes its results between candidate runs makes the comparison harder to interpret. Use a versioned retrieval snapshot or a controlled tool fixture when isolating a prompt change. When evaluating the live integration itself, record that it is a different kind of test.

A fake tool should enforce the same important input contract and return realistic failures. Returning success for every call removes the part of the problem we need the agent to handle.

Use fixtures to isolate behaviour, then retain separate integration checks for permissions, network errors and actual service compatibility.

A timeout, exhausted step budget or invalid structured response is a result of the test, even if no final answer exists. Store its failure category and the work completed before it stopped.

If a chart computes answer quality only for completed runs, show completion coverage beside it. Decide which failure categories automatically fail a task rather than letting an aggregation silently omit them.

The unit being tested is the assigned task. A missing answer does not remove that task from the workload.

A Python dictionary keeps each case’s expected outcome beside its observed outcome. Tool lists remain lists because a run may call several tools, including the same one more than once. A set is useful for membership checks against the allowed names.

The complete program below checks permissions on one case. The challenge adds outcome checking across a list of cases and validates the supplied record structure.

An empty call list passes this narrow permission check. It may still fail the task because no useful work was done.

case = {
    "allowed": ["search_manual", "propose_job"],
    "calls": ["search_manual", "create_job"],
}
allowed = set(case["allowed"])
violations = [name for name in case["calls"] if name not in allowed]
print(violations)

['create_job']

Separate cases with missing manuals, ambiguous equipment IDs and approval requirements. These groups are slices of the evaluation dataset. A change can improve ordinary lookups while making the small approval slice worse.

Report the number of cases in each slice. A perfect rate on one unusual case is weak evidence, and repeating that same case with slightly different wording does not necessarily add an independent situation.

Illustrative ordinary cases: 18 passing / 18 tested = 1.0
Illustrative approval cases: 0 passing / 2 tested = 0.0
Combined rate: 18 / 20 = 0.9

The combined rate hides a complete failure on the approval cases. Whether that blocks a release depends on the explicit contract, not on averaging it away.

Use development cases while changing prompts and tools. Keep a separate held-out set for checking the selected candidate. Split related conversations and documents together where needed so near copies do not appear on both sides.

Once a held-out failure becomes a debugging example, that case is no longer untouched evidence for the next change. Retain it as a regression case and refresh the independent evaluation process deliberately.

A private test set can still leak through repeated manual inspection. The information boundary applies to people tuning the system as well as to its prompt.

A model can choose different actions on repeated attempts. Run a case more than once when that variability matters, retaining the repetition number and configuration. Report both the case-level pattern and the individual failures.

Ten executions of one question help estimate variability on that question. They do not provide the same coverage as ten different maintenance situations. Compare candidates on the same case set and use a consistent repetition plan.

Seeds and fixed settings can help reproducibility where supported, but they do not promise identical behaviour across every provider or infrastructure change.

The SDK can execute a task over local dataset items and attach evaluator results. This example deliberately uses recorded outputs so we can verify the evaluation plumbing before spending money on model calls.

The task reads only the case ID. The evaluator receives the expected outcome separately. Replacing the lookup with an agent runner later should preserve that separation and should include the recorded trajectory in the returned result.

The experiment reports on these fixtures. It is not a benchmark of a live model.

from langfuse import Evaluation, get_client

recorded_outputs = {"missing-manual": {"outcome": "abstain", "calls": ["search_manual"]}}
data = [{"input": {"case_id": "missing-manual"},
         "expected_output": {"outcome": "abstain", "allowed": ["search_manual"]}}]

def task(*, item, **kwargs):
    return recorded_outputs[item["input"]["case_id"]]

def evaluate_contract(*, output, expected_output, **kwargs):
    outcome_ok = output["outcome"] == expected_output["outcome"]
    tools_ok = all(name in expected_output["allowed"] for name in output["calls"])
    return Evaluation(name="task-contract-v1", value=float(outcome_ok and tools_ok))

client = get_client()
result = client.run_experiment(
    name="maintenance-recorded-fixtures", data=data,
    task=task, evaluators=[evaluate_contract],
    metadata={"application_revision": "fixture-1", "dataset_revision": "fixture-1"},
)
print(result.format())
client.flush()

Give the evaluator an accurate outcome accompanied by an unapproved call. Then give it permitted calls with the wrong outcome. Both should fail. Also include a valid abstention with no write operation so the evaluator does not mistake restraint for failure.

These are tests of the evaluator itself. If it cannot reject a known bad record, a green dashboard is not useful evidence about the agent.

Botie’s changed workshop case keeps the answer label correct while introducing an unauthorised action. The repair must preserve the outcome check and add the missing permission check.

A small regression suite can show that known cases still work. It does not enumerate every possible instruction, manual revision or sequence of tool failures. Extend coverage from real problems and planned stress cases, while protecting sensitive production data.

Some properties need tests outside the language-model loop. Access control should be enforced by the service, and a duplicate write should be prevented by a transaction or idempotency rule rather than merely discouraged by a prompt.

The tool-name check in this lesson is deliberately incomplete: search_manual can be permitted while its requested organisation or equipment record is forbidden. Add executor-level tests for those arguments and inspect the resulting state. Include useful allowed cases near each forbidden boundary so a repair that refuses everything cannot appear successful.

The evaluator checks evidence of behaviour. The application still needs boundaries that remain effective when the model chooses badly.