What Can We Test Without Calling a Model?

We rename a tool argument and accidentally pass the wrong asset to the manual service. Running the real model ten times will not make that wiring clearer. We need a test that supplies a known call and checks exactly what our program does with it.

Then we need a different kind of test: will the intended model choose the correct tool on an unfamiliar request? Both matter. Keeping them separate makes a failed test much easier to investigate.

Join expected and observed calls by case ID. Keep missing runs visible.

{
  "passed": 2,
  "failed": 0,
  "missing": 0
}

Our program sends messages and tool definitions to a model, receives a proposed call, validates it, and invokes a service. A test double can replace the model response. A separate fake can replace the service.

Replacing the model alone does not make tool effects harmless: framework test models may still call real tool functions. Inject isolated dependencies and block unexpected network requests in the test environment.

Choose a known fixture at each external boundary. Never let a local test accidentally create a production work order.

For this exercise, each case expects exactly one manual-read call. We compare its tool name and asset. In a real test, also inspect identity, authorisation decisions and the returned observation. A correct-looking final answer cannot establish these facts.

If several valid routes can solve the task, do not require one exact sequence unless that sequence is itself a requirement. Strict trajectory equality can fail useful alternatives.

Our single-call contract is intentionally narrow. A full trajectory requires call IDs, ordering constraints and effect checks.

wanted = {"tool": "read_manual", "asset": "P7"}
got = {"tool": "read_manual", "asset": "P8"}
print(wanted["tool"] == got["tool"])
print(wanted["asset"] == got["asset"])

True
False

TestModel is a procedural stand-in, not a small language model. The following smoke test has no tools and returns a supplied string. It establishes that the application can run an agent and read its output through the framework API.

For a tool test, override the model on your existing agent and supply fake dependencies. TestModel normally exercises tools with generated arguments; FunctionModel gives finer control when a particular sequence matters. Read the recorded calls rather than assuming a final string proves the calls were correct.

The recipe runs in your own Python environment. The browser challenge below tests a plain-Python report over recorded calls.

from pydantic_ai import Agent, models
from pydantic_ai.models.test import TestModel

models.ALLOW_MODEL_REQUESTS = False
agent = Agent(TestModel(custom_output_text="Manual lookup needs an asset."))
result = agent.run_sync("Which manual applies?")
assert result.output == "Manual lookup needs an asset."
print(result.output)

Use a mock chat model when testing application reactions to a known response. If the agent uses LangGraph persistence, use an isolated checkpointer and a distinct thread for each test unless the test deliberately exercises resumption. Otherwise an earlier run can supply state that hides a missing step.

A small provider integration suite can then exercise tool schema compatibility and message formatting. A task evaluation tests the assembled system on representative requests. Record provider errors separately from scored answers.

The earlier framework lessons show the actual agent and graph APIs. These testing boundaries apply whichever framework owns the loop.

Expected cases form the denominator. Match observed calls by ID and mark absent results as missing. Reject duplicate IDs because there is no unambiguous call to compare under this one-call-per-case contract.

The same rule applies when workers finish out of order. Joining two lists by their positions can compare P7’s expected call to P8’s actual call and create misleading failures.

An empty batch reports zero passes, zero failures and zero missing results. It provides no acceptance evidence.

Keep the wrong-asset call as a failing regression fixture. Change the application mapping, rerun that fixture, then try another asset and a missing result. This tests the repair against the rule rather than only the particular name P7.

Finally run the intended model on new task cases. A deterministic test can establish that a valid call is handled properly; it cannot establish how often the model proposes one.

Keep unit, integration and task-evaluation results separately named in CI so a reader knows what passed.