A Better Score on Which Dataset?

Botie lists several training runs and chooses the smallest error. The new run looks excellent until we notice that it used a different validation table. Both tables happen to be at version four. Another score was measured on the final test set. Sorting those numbers together does not tell us which model improved on the same evidence.

We need a run record that connects the model to its inputs and assessment. Then we can decide which records belong in this particular comparison before choosing an artifact.

A tempting low score came from a different validation table.

{
  "runId": "C",
  "modelUri": "models:/C",
  "trainingTable": "lab.training.calibrated_features",
  "trainingVersion": 4,
  "validationTable": "lab.validation.sensor_readings",
  "validationVersion": 4,
  "split": "S"
}

Think of a run as the record of one execution of our experiment. It keeps what we tried beside what happened. The run ID names that execution. The saved model file is an artifact, meaning an output we keep, and its URI is the address another program can use to request it.

The input needs an address too: record the training table identity beside its version, and do the same for validation. Another table can have its own unrelated version four. The split says which assessment rows were used inside the selected validation snapshot. A dictionary keeps those meanings attached to the score and model.

Throughout this example the metric is validation mean absolute error in the same temperature unit, and a smaller value is better.

Declare the validation table identity, its version and the split used for this selection. Exclude records with different assessment evidence and records marked as final test results. Only then rank the remaining candidates by validation MAE.

Candidates may deliberately use different training tables, training versions or parameter settings. That can be the experiment we want to run. Hold the assessment evidence fixed while recording those training differences so we can investigate what changed.

Repeatedly selecting on the validation set can still overfit that selection process. The final untouched assessment remains separate.

runs = [{"runId":"a","validationTable":"lab.validation.easy_readings","validationVersion":4,"split":"S"}, {"runId":"b","validationTable":"lab.validation.sensor_readings","validationVersion":4,"split":"S"}]
print([row["runId"] for row in runs if row["validationTable"] == "lab.validation.sensor_readings" and row["validationVersion"] == 4 and row["split"] == "S"])

['b']

An MLflow dataset record can describe where rows came from and include a compact digest, a summary used to recognise that dataset. The record is useful for inspection, but it does not contain all the source files. Keep the named Delta table and version, with a retention plan that allows the needed read.

Training may transform those rows before the model sees them. Record the transformation code and settings as well as any preprocessing learned from training data. Otherwise two runs could name the same raw table while training on different representations.

This is why the earlier lineage work matters: we want to follow a model back through the actual operations that produced its input.

The reference constructs dataset records from explicitly versioned training and validation Delta reads. It selects the named validation split before logging that input. The supplied fitted model and measured validation MAE must come from that exact declared procedure.

The input example omits the label and must match the model’s actual input representation. Use the returned model_uri rather than inventing an artifact path from a displayed metric. Keep the runtime and dependency versions with the project.

The tracking run now names both tables, their versions and the assessment split beside the model artifact.

import mlflow
import mlflow.sklearn

training_table, training_version = "lab.training.sensor_features", 4
validation_table, validation_version = "lab.validation.sensor_readings", 8
validation_split = "S"
training = spark.read.option("versionAsOf", training_version).table(training_table)
validation = spark.read.option("versionAsOf", validation_version).table(validation_table)
validation = validation.filter(validation["split_id"] == validation_split)
training_dataset = mlflow.data.from_spark(
    training, table_name=training_table, version=str(training_version), targets="target",
)
validation_dataset = mlflow.data.from_spark(
    validation, table_name=validation_table, version=str(validation_version), targets="target",
)
with mlflow.start_run() as run:
    mlflow.log_input(training_dataset, context="training")
    mlflow.log_input(validation_dataset, context="validation")
    mlflow.log_param("training_table", training_table)
    mlflow.log_param("training_table_version", training_version)
    mlflow.log_param("validation_table", validation_table)
    mlflow.log_param("validation_table_version", validation_version)
    mlflow.log_param("validation_split", validation_split)
    mlflow.log_metric("validation_mae", float(validation_mae))
    model_info = mlflow.sklearn.log_model(
        fitted_model, name="model", input_example=input_example,
    )
    print(run.info.run_id, model_info.model_uri)

Once a comparable run is selected, return a manifest containing its run ID and model URI together with both training and validation table identities, their versions and the split. Keeping version numbers without their table identities would leave the next consumer guessing which histories to read.

A real release also needs input/output contracts, runtime dependencies and approval evidence appropriate to the application. Loading an artifact must follow the model-file security boundary from the other course area.

The manifest should let another piece of software identify the artifact and its declared evidence.

If no run used the required validation table, version and split, the result is no eligible comparison. Raise a clear error instead of choosing the smallest number from another dataset. In our challenge, exact metric ties use run ID order only to make the output deterministic.

That tie-breaker is not evidence of statistical superiority. Small observed differences may need paired analysis and uncertainty estimates before a practical conclusion can be defended.

Keep that complete chosen record on the wire all the way to the manifest.

A useful fresh case gives a run on another validation table an extremely good number while leaving its version and split unchanged. If it still wins, part of the evidence identity was lost. Other cases change the version or split.

Keep final-test rows out of this selection altogether. Looking at a test score to choose a candidate spends the independence that test set was meant to provide.

Can the job identity read the table named in this manifest?