Section 22 of 24 · 12 lessons

Prompting, Evaluation and Iteration

Botie answers the first maintenance question correctly, then gets the next one wrong. Before rewriting the prompt, we need to understand what happened. Choose useful examples, follow the tool calls through their traces, and build tests that expose the failure. We will use Langfuse and other evaluation tools, compare candidate revisions, and carry a repair through a fresh shift without losing the behaviours that already worked.

0 of 12 finished

Teaching the Task Through Examples

An example can accidentally teach Botie to recommend an inspection every time. Choose demonstrations that show why the right response changes with the evidence.

  1. 01Which Examples Should We Put in the Prompt?Botie keeps making an inspection sound mandatory. Show what a supported answer looks like, then check that the examples teach the distinction we intended.
    Build with Botie →Read the lesson

Seeing What Happened

Botie returned the wrong answer, but which step went wrong? Record the work as it happens so we can distinguish a missing manual from a failed tool or an unsupported answer.

  1. 02Why Did Botie Take So Long?An answer arrives late, but the text cannot tell us where the time went. Follow one maintenance request through retrieval, tool calls and the final response.
    Build with Botie →Read the lesson
  2. 03Can We Find the Run Behind This Score?A dashboard says answers improved. Connect scores to their requests and prompt versions so we can see which answers were actually checked.
    Build with Botie →Read the lesson

Building Evaluation Cases

Keep the request and its expected outcome together. The next prompt revision needs cases that can reveal a mistake, including cases we did not use to design it.

  1. 04What Would Count as Doing the Job?A plausible answer can still come from an unacceptable sequence of actions. Build maintenance cases that check the answer, permitted tools and failure behaviour.
    Build with Botie →Read the lesson

Testing the Program Around the Model

A model can vary its wording while our code keeps passing the wrong asset to a tool. Separate reproducible application checks from evaluations of model behaviour.

  1. 05What Can We Test Without Calling a Model?An agent framework can run our tools, but a test model and a real model answer different questions. Build contract checks before spending time on live evaluations.
    Build with Botie →Read the lesson

Reviewing Answer Quality

Some requirements have exact checks. Others need judgement. Define what the reviewer should look for and test whether those judgements are dependable.

  1. 06Who Checks the Model That Checks Botie?Some answers need interpretation rather than an exact comparison. Give a judge a clear rubric, then test its judgements against evidence and human review.
    Build with Botie →Read the lesson

Finding a Change Worth Making

The answer is wrong. Was the right evidence available? Trace the failure to a specific question, then change one part and inspect what improves or regresses.

  1. 07Where Did the Answer First Go Wrong?Several answers are wrong, but they did not fail for the same reason. Inspect the evidence and tool results before rewriting the prompt.
    Build with Botie →Read the lesson
  2. 08Did the New Examples Cause the Improvement?A candidate improves after we change its model and prompt. Separate the changes, pair the task results, and inspect what became worse.
    Build with Botie →Read the lesson

Deciding Whether to Release

The candidate looks promising on our examples. Check the remaining requirements before allowing it to handle real requests.

  1. 09Is This Version Ready for the Next Shift?A new agent can improve average answers while introducing an unacceptable failure. Compare the same cases and keep critical behaviour and operating budgets visible.
    Build with Botie →Read the lesson

Testing and Operating the Boundaries

A guardrail that blocks every request would also stop useful work. Test ordinary jobs alongside attempts to misuse the system, then practise responding when a released version crosses a boundary.

  1. 10Can We Make Botie Cross the Boundary?A guardrail passes our friendly examples. Try to break a specific boundary, inspect the actual effects, and check whether ordinary maintenance questions still work.
    Build with Botie →Read the lesson
  2. 11What Happens When the New Shift Finds a Failure?The released assistant attempts an unapproved job. Stop further effects, account for work already in flight, and turn the failure into evidence for a careful restart.
    Build with Botie →Read the lesson

A Repair From Failure to Fresh Evidence

A stale manual trips up the assistant. Carry that failure through a tool repair, preserve useful behaviour and test the candidate on a new shift.

  1. 12Repair Botie’s Maintenance AssistantFollow a stale-manual failure through a test, a repair and a fresh evaluation. Keep unsupported answers and unapproved actions out of the release decision.
    Build with Botie →Read the lesson