← All workshopsRead Can We Find the Run Behind This Score?

Prompting, Evaluation and Iteration

Keep the missing evaluations visible

BOTIE

Select the requested rubric revision and join its scores to the run records by traceId. Preserve a missing score as null.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Select the requested rubric revision and join its scores to the run records by traceId. Preserve a missing score as null.

Botie wants to compare prompt revisions, but scoring finishes out of order and some runs have no judgement.

Inspect the component calculation
Attach scores by trace ID(
  Run records: Recorded runs and prompt revisions
  One evaluator version’s scores: Keep matching records(
    Records: Trace scores and rubric revisions
    Required field value: Requested rubric revision
    Field to match: rubric
  )
)
Examples and results
Recorded runs and prompt revisions
[
  {
    "traceId": "A",
    "promptVersion": 3
  },
  {
    "traceId": "B",
    "promptVersion": 3
  },
  {
    "traceId": "C",
    "promptVersion": 2
  }
]
Trace scores and rubric revisions
[
  {
    "traceId": "B",
    "value": 1,
    "rubric": "grounded-v2"
  },
  {
    "traceId": "C",
    "value": 1,
    "rubric": "grounded-v2"
  },
  {
    "traceId": "A",
    "value": 0,
    "rubric": "grounded-v2"
  }
]
Requested rubric revision
grounded-v2
Required result
[
  {
    "traceId": "A",
    "promptVersion": 3,
    "score": 0
  },
  {
    "traceId": "B",
    "promptVersion": 3,
    "score": 1
  },
  {
    "traceId": "C",
    "promptVersion": 2,
    "score": 1
  }
]
Recorded runs and prompt revisions
[
  {
    "traceId": "A",
    "promptVersion": 4
  },
  {
    "traceId": "B",
    "promptVersion": 4
  }
]
Trace scores and rubric revisions
[
  {
    "traceId": "A",
    "value": 1,
    "rubric": "grounded-v1"
  },
  {
    "traceId": "B",
    "value": 1,
    "rubric": "grounded-v1"
  },
  {
    "traceId": "A",
    "value": 0,
    "rubric": "grounded-v2"
  },
  {
    "traceId": "B",
    "value": 0,
    "rubric": "grounded-v2"
  }
]
Requested rubric revision
grounded-v2
Required result
[
  {
    "traceId": "A",
    "promptVersion": 4,
    "score": 0
  },
  {
    "traceId": "B",
    "promptVersion": 4,
    "score": 0
  }
]
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

These simplified records allow one score per trace for the selected rubric. Real systems also distinguish observation IDs, evaluator sources and repeated runs. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.

A new rubric has scored only one of the new version’s runs

This check uses the pieces from “Read the version’s score beside its coverage”.

BOTIE

Report prompt version 5 using grounded-v3. Older-rubric results are present, and one selected run is still unscored. Keep a low fractional score distinct from missing evaluation.
Recorded runs and prompt revisions
[
  {
    "traceId": "night-A",
    "promptVersion": 5
  },
  {
    "traceId": "night-B",
    "promptVersion": 5
  },
  {
    "traceId": "night-C",
    "promptVersion": 4
  }
]
Trace scores and rubric revisions
[
  {
    "traceId": "night-A",
    "rubric": "grounded-v2",
    "value": 1
  },
  {
    "traceId": "night-B",
    "rubric": "grounded-v2",
    "value": 1
  },
  {
    "traceId": "night-C",
    "rubric": "grounded-v3",
    "value": 1
  },
  {
    "traceId": "night-A",
    "rubric": "grounded-v3",
    "value": 0.25
  }
]
Requested prompt revision
5
Requested rubric revision
grounded-v3

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why does a low score belong in the mean while a missing score needs a coverage report, and what would make two rubric revisions incomparable?

Keep run identity, prompt revision and evaluator revision attached to scores. Missing evaluation is a distinct state that a mean alone cannot explain.

These simplified records allow one score per trace for the selected rubric. Real systems also distinguish observation IDs, evaluator sources and repeated runs.

Work through the Python challenges →
Continue the courseWhat Would Count as Doing the Job? →
← Why Did Botie Take So Long?