Data Pipelines for Learning
A training table is the result of decisions about records. Follow Botie from unequal input files to historical features, rejected sensor batches and late events. Keep enough history to explain which data actually produced a model.
Building the training table
One sensor file has fewer rows than another. Before combining them, we need to know what each row represents and what information was available at the time.
- 01Ready to exploreOne Training Dataset, Several Unequal Files
Merge sufficient statistics instead of averaging the averages of unequal partitions.
- 02Ready to exploreRebuild What the Model Could Have Known
Choose the latest eligible feature using both event time and availability time.
Repairing a changing pipeline
The next file uses a different column layout, or a reading arrives after its reporting window. Work out what the existing procedure would do before deciding how to change it.
- 03Ready to exploreStop a Broken Sensor Batch Before It Reaches Training
Separate row validation from batch acceptance and preserve rejected evidence for investigation.
- 04Ready to exploreThe Reading Happened Before the Window Closed
Assign event-time windows and make late-data policy explicit before updating an online feature.
- 05Ready to exploreWhich Inputs Produced This Model?
Bind reusable artifacts to input versions and code versions so a changed dataset invalidates the right work.