Which Records Should Botie Bring into the Conversation?

A technician asks about a seal on P-2. Botie has maintenance manuals for several sites, years of service tickets, and a limited amount of text it can send in one model request. Copying everything would include irrelevant records and perhaps records this technician is not allowed to read.

We need to choose what enters the conversation. This means finding useful material, respecting permissions, and preserving enough of its identity to check the eventual answer.

Filter candidate passages by asset permission before taking the ordered prefix that fits. Try a passage that consumes the remaining room exactly.

Retrieval produces candidates likely to help answer a question. A lexical search can match words; an embedding search can compare representations; a reranker can reorder candidates using another scoring method. These methods have different failure cases. None automatically knows every application permission or guarantees that a passage answers the question.

Our inputs begin after ranking. We receive records containing an ID, an asset, text and a supplied token cost. Selection decides which permitted records reach the model. Keeping those stages distinct lets us ask whether a failure came from missing candidates or from discarding useful candidates later.

A relevant-looking passage about a different pump can be dangerous if the asset identity is lost during selection.

The authenticated caller’s scope determines which records are eligible. Apply access restrictions during retrieval where supported, and enforce them before selected content reaches the model or any externally visible log. A prompt saying “do not reveal restricted manuals” is too late if the service already sent the restricted text.

In this example, a record is eligible when its asset occurs in an authoritative allowed list. A real document system may require organisation, role, document version and operation-specific checks. Asset membership is our deliberately small permission model.

The workshop lets you inspect the candidate records locally. Its fixtures contain no private documents; they illustrate where an application would enforce the boundary.

A model request contains more than excerpts. Instructions, the question, tool schemas and conversation messages all consume context. The reply may need an allowance as well, depending on the model’s limit semantics. Start by measuring or reserving those parts, then decide how much remains for evidence.

Our invented accounting uses one total window and explicit input and output reserves. Costs are supplied whole-token counts; word count is not token count. Message wrappers and formatting can add overhead, so a production implementation should count the final serialised request with the correct model tokenizer and leave margin where needed.

Total allowance = 20 tokens
Reserved non-evidence input = 8 tokens
Reserved output = 4 tokens
Evidence allowance = 20 − 8 − 4 = 8 tokens

If reserves exceed the total, reject the configuration or deliberately shorten something. Do not silently call the remaining budget zero and claim the request fits.

After filtering, our rule keeps complete records in ranked order until the next record would exceed the available budget. It stops at that point. This preserves a prefix of the ranking and avoids chopping a sentence in half.

It can waste room. If the first permitted passage is too large, a shorter passage later in the list will not be selected. Skipping oversized passages, compressing them, or changing chunk sizes are different policies with their own tradeoffs. State which policy you use so a test can distinguish an intended choice from a bug.

Permitted passage costs = [6, 5]
Evidence allowance = 8
Take the first passage: used = 6
The next total would be 6 + 5 = 11
11 exceeds 8, so stop

The result contains one source. We have made a budget decision, not shown that the remaining source is sufficient to answer the question.

costs = [3, 8, 2]
budget = 7
used = 0
selected = []
for position, cost in enumerate(costs):
    if used + cost > budget:
        break
    selected.append(position)
    used += cost
print(selected)
print(used)

[0]
3

A passage should retain its source ID, document version and useful location such as a page or section. If it is split, keep the chunk-to-document mapping. If it is summarised, retain the underlying evidence and make the summary’s status clear. Otherwise an answer can name a document without giving anyone a way to check the relevant text.

A citation ID proves only that the answer points somewhere. Check whether the cited material supports the claim, refers to the right asset, and was available to the user. An outdated manual can be perfectly cited and still inappropriate for the current machine.

The workshop outputs IDs from the selected records. Read the corresponding text in the input records to inspect what the model would actually receive.

A service ticket may contain an instruction directed at the assistant. A manual page might even include a quoted conversation. Keep that text labelled as evidence, as we did in the prompt lesson. It must not change the application’s tool permissions or approval state.

Retrieval makes prompt injection a practical data-flow problem: content from outside the application enters the model’s input. Restrict available tools and validate their calls regardless of how convincing the generated explanation sounds. If a task only needs an answer from manuals, do not grant it a general-purpose write tool.

Encoding the document cleanly helps inspection, but it does not establish that a model will respect the data boundary.

Conversation history can consume the same room as evidence. An old summary may omit a qualification or carry forward an earlier mistake. Keep durable facts separate from provisional model statements, and preserve source links for facts that need verification.

A short-lived conversation state is different from a long-term user memory. Decide which values should persist, who may see them, how corrections replace stale values, and when they expire. Loading every past message is a memory policy, even if it was chosen accidentally.

For this maintenance task, the asset ID and current question matter. An unrelated ticket from last week should not displace the manual excerpt needed now.

When an answer fails, inspect the selected records. Was a relevant passage retrieved? Was it authorised? Did it fit? Did the selected text actually contain the needed fact? A prompt change cannot recover evidence that the request never contained.

Evaluate retrieval coverage on labelled questions separately from answer support. Include a case where no permitted document answers the question. The correct system behaviour may be to explain the gap or request more information, rather than invent a plausible maintenance interval.

A later observability lesson will record these stages. Here the selected IDs make the budgeting consequence visible without a live model.