Does Botie Need to Choose the Next Step?

A technician asks for the inspection date of pump P7. We already know the steps: check access, look up P7, and display the date. Giving Botie freedom to invent a plan adds decisions to a job whose route we can write down ourselves.

Now consider a request to explain why several pumps began vibrating after a maintenance visit. The useful follow-up depends on what the records reveal. We need a way to decide where a model’s choice of next step actually helps. We will compare candidate controllers on the same invented maintenance requests, including their mistakes.

Keep the task checks beside each controller’s time and cost. Connect the comparison before choosing the amount of autonomy.

workflow

Start with the result the technician needs. For a record lookup, success means returning the correct authorised record or accurately reporting that it is unavailable. For an investigation, success may mean identifying relevant evidence and explaining what remains uncertain. A fluent response that omits the missing evidence does not complete that job.

Choose the acceptance checks before trying architectures. Keep cases where a record is missing, where access is denied, and where two assets have similar names. If only the happy path appears in the test set, the most fragile controller can look convincing.

The checks describe the job. The controller is one proposal for doing it.

A direct database lookup is a useful baseline for a date request. A baseline is a working comparison that tells us what extra machinery improves. It may be ordinary code, a search interface, or one model call with supplied evidence.

A fixed workflow can validate a request, retrieve a manual, ask a model for a supported summary, then check its output. Branches for absent evidence and failed validation remain explicit. We can test each branch without asking a model to discover the order again. The wording step may still be uncertain, so fixed orchestration does not make the whole system deterministic.

Give the baseline a fair implementation. An intentionally weak prompt would make a larger architecture appear more useful than it is.

Suppose the first maintenance record points to a replacement coupling. Botie might need the coupling specification next. A different record might point to a changed sensor. We can allow a model to choose among a limited set of read operations based on that evidence.

This creates a new responsibility: checking whether each proposed action is permitted and whether the investigation has enough evidence to stop. Tool feedback tells the loop what happened. A missing record can trigger a supported answer about uncertainty rather than another speculative query. Keep the earlier output and permission checks when adding this freedom.

Choose an agent because the task benefits from adaptive decisions. The presence of several tools alone does not establish that benefit.

Separate specialists can examine genuinely independent records in parallel. Before adding them, identify what each receives, what it must return, and who combines conflicting findings. A handoff should preserve the request identity and source IDs. The coordinator needs a way to recognise partial or failed work.

Sending the same full conversation to several agents repeats input work and introduces several proposals to reconcile. Parallel work can reduce elapsed time while increasing total compute. A copied mistake may appear in every answer, so agreement is not independent evidence. Begin with one controller and measure whether delegation improves the task enough to justify its coordination and operating costs.

For the maintenance desk, two independent manual searches may be useful. Two agents that both create the same work order introduce a different problem.

The workshop stores each controller as a dictionary. passed is a list of boolean outcomes in the agreed case order. latency_ms and cost_units have one entry for each of those same cases. rank records our declared preference for less autonomy: rules first, then workflow, then agent. A rank is a design policy, not a measure of intelligence.

We summarise how many checks passed, the slowest observed request, and total cost over the fixture. Comparing different case sets would confound architecture with task difficulty. Real evaluations should retain case IDs and versions, repeat stochastic runs, and report uncertain estimates. Our short list assumes that alignment has already been checked.

Latency and cost answer different questions. A parallel controller may have a smaller wall-clock maximum and a larger total bill.

passed = [True, False, True]
latency_ms = [420, 510, 470]
cost_units = [2, 3, 2]
assert len(passed) == len(latency_ms) == len(cost_units)
for case, (ok, elapsed, cost) in enumerate(zip(passed, latency_ms, cost_units), start=1):
    print(case, ok, elapsed, cost)
print("passes:", sum(passed))

1 True 420 2
2 False 510 3
3 True 470 2
passes: 2

Filter the summaries against the acceptance requirements. Our fixture requires every supplied task check to pass and bounds observed latency and total cost. Among candidates that meet those requirements, choose the lowest autonomy rank. If ranks tie, choose lower total cost, then the alphabetically earlier name so the result is repeatable.

This is an explicit teaching policy. A real service may prioritise tail latency, audit requirements, operating effort or a risk threshold differently. State that policy before looking at the candidate scores. If no candidate passes, return a review outcome. Quietly selecting the least bad failure would erase the meaning of the requirements.

eligible = task_passes >= required_passes
           and worst_latency <= latency_limit
           and total_cost <= cost_limit

Passing this fixture is permission to investigate a candidate further. It is not a guarantee about future requests.

The changed workshop case gives the fixed workflow a request it cannot resolve. Its score now fails the task requirement, while the agent candidate meets the same requirements. Keep the selection policy unchanged and inspect which evidence changed the decision.

A single adaptive example should not make every request autonomous. Routing well-defined requests through the simpler path and reserving an agent for measured difficult cases can be useful. The router needs its own evaluation, including false escalations and difficult requests it incorrectly keeps on the simple path.

The useful outcome is a design tied to task evidence, with a clear fallback when the evidence is insufficient.

A workflow and an agent can use the same scoped read tool, output validator and human approval boundary. Changing who chooses the next step does not change who is authorised to read a customer’s records or create an inspection.

Version the controller along with prompts, tools, model settings and evaluation data. Re-run the comparison after a meaningful change. Watch the real request mix: a controller chosen for mostly date lookups may need reconsideration when most users begin asking open investigations. Frameworks in the next lessons can organise these components, but their abstractions should leave the decision boundaries visible.

We can now ask a more useful framework question: does it make the controller we chose easier to implement, test and inspect?