Which Inputs Produced This Model?

Track a Run’s Inputs and Outputs

BOTIE

A model file can outlive the job that created it. When a prediction changes later, the filename alone tells us very little. Which source records were read? Did preparation code change, or did somebody replace a file in place? Answering those questions becomes harder when each pipeline tool records its work differently.

OpenLineage brought projects including Airflow and dbt into a common effort to describe these relationships. Its September 2021 initial release included a shared specification and integrations. The model distinguished a job, which describes work to do, from a run, which is one execution, and identified the datasets involved. That lets information from separate tools describe a connected processing history.

We will start with a much smaller record: the exact dependencies of one prepared feature file. This record is a manifest. Comparing its input and code identities lets us explain why a previous result can be reused, or why replacing an input requires doing the calculation again.

Sources for this history

The Cache Returned a Perfectly Good Model for the Wrong Data

Botie caches prepared features to avoid repeating expensive work. The cache key is a filename. Someone replaces the file’s contents, and the next run reuses yesterday’s features. The workflow graph is correct, but the identity of its inputs is not.

A reusable artifact needs a description of the exact inputs and transformation that produced it. That description also gives us a way to investigate a bad model later.

These supplied IDs stand for immutable versions. This comparison neither hashes artifacts nor verifies publisher signatures.

Step 1 of 1

A manifest makes a cache decision inspectable

A changed dependency requires a different artifact identity.

The unchanged code does not make the changed input compatible.
Dependency	Stored	Requested
Input version	data-v1	data-v2
Preparation code	prep-v3	prep-v3

Input version
Stored
data-v1
Requested
data-v2
Preparation code
Stored
prep-v3
Requested
prep-v3

The unchanged code does not make the changed input compatible.

Separate location from identity

Reuse requires agreement on every declared dependency.

Invalidate the stale feature artifact

A new input dataset must not reuse a feature file produced from an older version.

BOTIE

A new input dataset must not reuse a feature file produced from an older version.

Choose an example

Cached input version

Requested input version

Cached code version

Requested code version

Whether both input and code versions match

true

Follow the operations

1. Cached input version
```
d1
```
2. Requested input version
```
d1
```
3. Compare complete values
```
true
```
4. Cached code version
```
c1
```
5. Requested code version
```
c1
```
6. Compare complete values
```
true
```
7. Require both checks
```
true
```
8. Result
```
true
```

These supplied IDs stand for immutable versions. This comparison neither hashes artifacts nor verifies publisher signatures.

1. Separate location from identity

A path tells us where to look. A content digest is an identifier computed from the file’s bytes. Changing those bytes changes the digest except in the rare case of a hash collision. A trusted digest or an immutable version lets us check which input we found. Keep human-readable metadata beside it so an investigator can understand the dependency.

A manifest lists the input versions, code version, configuration and schema that define the transformation.

Do not include credentials or raw personal data in a public manifest. An identifier may itself be sensitive.

2. Every required dependency must match

For our small example, compare input version and code version separately. Both must match before using the cached result. A full key includes all behaviour-changing configuration as well.

input matches = 0
code matches = 1
cache reusable = 0 AND 1 = 0

Equal versions are meaningful only if the version identifiers are immutable and trustworthy.

Serialise the meaning rather than insertion order

The manifest is a dictionary of declared dependencies.

Python

import json
manifest = {"data":"v1", "code":"v2"}
print(json.dumps(manifest,sort_keys=True,separators=(",", ":")))

Output

{"code":"v2","data":"v1"}

The challenge hashes these deterministic bytes. It does not infer dependencies that were omitted from the manifest.

Check yourself

What belongs in an artifact identity?

Question 1 of 2

May this cache entry be reused?

Use the information below.

Stored input v1, requested input v2; code version unchanged.

3. Canonicalise before hashing

Two dictionaries can contain the same manifest entries in different insertion orders. Serialising with sorted keys gives a deterministic representation for our JSON-compatible manifest. Hash its UTF-8 bytes.

A digest detects a difference in bytes. It does not prove who published those bytes or whether they are safe to load.

For a production format, specify canonicalisation, numeric representations and versioning explicitly rather than assuming every serializer agrees.

4. Use lineage when the model fails

Follow the served model to its training run, prepared dataset and raw input versions. This can distinguish a new training bug from a changed producer. Record failed runs too, without labelling their partial artifacts as complete.

If a source dataset is withdrawn, lineage helps locate affected artifacts. Deleting a manifest alone does not remove data from every copy or a trained model.

The workshop checks two dependencies; the Python manifest exercise extends the identity to arbitrary nested configuration.