Give Each Prediction the Features It Could Actually Have Seen

Botie can read the approved table. He now joins the latest temperature to every old inspection. Unfortunately, a temperature correction published after an inspection becomes an input to that inspection’s prediction. The training example has learned something from its own future.

We need a feature value belonging to that pump and available by that prediction time. We will also decide how old a feature may be. A missing value must remain visible rather than being quietly replaced by the newest reading.

The current table is permitted, but a later correction leaks into earlier training rows.

[
  {
    "pump": "P7",
    "at": 6,
    "temperature": 10
  },
  {
    "pump": "P2",
    "at": 6,
    "temperature": 20
  }
]

Keep the two kinds of records visible. A feature-history row names a pump, the minute its value was published and the value. A prediction row names a pump and the minute when a decision needed to be made. These timestamps describe different events.

For a prediction about P7, begin with P7’s feature history. Then ask which values had been published by the decision minute. A recent reading from another pump cannot fill this input just because its timestamp is convenient.

Both Python inputs are lists of dictionaries. The same pump can occur in several prediction dictionaries because it needs decisions at different times. Our invented shift uses integer minutes from one start time; real timestamp fields need consistent types and a clear time-zone convention.

Keep one result per prediction in its original order, including predictions with no usable feature.

An event timestamp says when something happened. A publication timestamp says when this particular computed feature became usable. A correction can refer to an old event while arriving after the prediction.

The workshop compares publishedAt with the prediction’s at field. Published history must not be backdated or overwritten in a way that invents earlier availability. An as-of API cannot recover the true arrival history if the table falsely labels a later correction as previously known.

Read the timestamp definition as an input contract: when did this computed value become usable?

First match the pump. Then keep features published no later than the prediction. Finally apply the declared maximum age. The workshop treats both timestamp boundaries as inclusive. Only after those checks do we choose the greatest remaining publication time.

If no candidate remains, return null for that prediction’s feature. A legitimate missing value is different from zero, and neither is permission to read a future row. If the latest available row explicitly contains null, keep that null instead of quietly searching backward for a nicer value.

prediction minute = 6
feature publication minutes = [3, 8]
maximum age = 5 minutes
3 <= 6 and 6 - 3 <= 5: eligible
8 > 6: unavailable
selected value = 10

A fresh sensor value can still be wrong; the earlier range and quality checks remain part of the pipeline.

history = [{"pump":"P7","publishedAt":3,"value":10},{"pump":"P7","publishedAt":8,"value":99},{"pump":"P2","publishedAt":4,"value":20}]
candidates = [row for row in history if row["pump"] == "P7" and row["publishedAt"] <= 6]
print([(row["publishedAt"], row["value"]) for row in candidates])

[(3, 10)]

A Delta snapshot identifies which committed contents of a table we read. A point-in-time lookup selects suitable feature rows inside those contents for each prediction timestamp. Both may be needed for an honest training reconstruction.

Record a feature-table snapshot and the lookup contract. If later backfills rewrite old feature rows, an unpinned latest-table read may change an old training set even though its prediction timestamps stay unchanged. Retained snapshots or immutable publication history prevent that change from being hidden.

A feature key identifies the entity. The timestamp key must be explicitly designated for time-series lookup; merely adding a timestamp to an ordinary equality key does not create as-of semantics.

The optional reference registers a time-series feature table and builds a training set from prediction rows. FeatureLookup names the entity and prediction-timestamp columns. Its lookback window provides a training-time age bound.

The feature history is assumed to use actual publication availability as its timestamp. Registering a column cannot prove that semantic claim. Check the selected runtime’s table constraints and API support, and do not use preview pipeline primary-key features unless you deliberately choose a preview environment.

Databricks distinguishes this historical join from online latest-value lookup. Do not assume a training lookback setting automatically enforces the same age policy in every online serving path.

from datetime import timedelta
from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup

fe = FeatureEngineeringClient()
fe.create_table(
    name="lab.training.published_temperatures",
    primary_keys=["pump", "published_at"],
    timeseries_columns="published_at",
    df=published_features,
)
lookup = FeatureLookup(
    table_name="lab.training.published_temperatures",
    feature_names=["temperature"], lookup_key="pump",
    timestamp_lookup_key="prediction_at",
    lookback_window=timedelta(minutes=5),
)
training_set = fe.create_training_set(
    df=predictions, feature_lookups=[lookup], label="target",
    exclude_columns=["pump", "prediction_at"],
)
training_rows = training_set.load_df()

Training can return a missing feature for an early prediction. Serving needs an explicit policy for that same condition: use a declared fallback, request another measurement or decline the prediction. Filling missing values differently in the two paths creates a new model procedure.

Publish and test the transformation version as well as feature values. A shared table name is insufficient if training calculates a five-minute window while serving uses an hour, or if one side clips readings and the other retains them.

Pass missing results to a declared fallback policy after this lookup.

Move only the prediction time across a publication boundary and predict when the selected value changes. Then shorten the maximum age so the previously eligible row becomes stale. Add another pump with a recent feature and make sure it never fills a missing value for the requested pump.

Conflicting values at the same entity and publication time are another data-contract failure. Stop rather than letting a distributed row order choose one. A fresh result should preserve which prediction is missing and why it could not use the later record.

When a new correction arrives, inspect its publication timestamp before letting it change a historical training row.