Databricks for ML Pipelines
Databricks gives a team a shared environment for processing data and building ML applications. Apache Spark is one of the tools that performs the calculations, with work spread across several machines. We’ll start with why that became useful, then examine what a training run needs to remember about its data and model files.
Following a distributed calculation
Spark can spread a calculation across workers. Begin with the instructions it receives, then look at why some records have to move between workers before a result can be calculated.
- 01Ready to exploreWriting the Steps Is Different from Running Them
A cleaning rule works on one file. What do Spark and Databricks add when a team needs to run that rule across many files, and when does the calculation actually happen?
- 02Ready to exploreOne Busy Pump Can Keep Every Worker Waiting
Most workers have finished, but one pump’s records keep the job running. Follow where matching records must meet before deciding how to share that work.
Keeping a dependable table history
The sensor records changed after our first training run. Which version should an old model still refer to? We need to preserve a consistent view of the table for each run.
- 03Ready to exploreWhich Version of the Table Did the Model Read?
The code stayed the same, but a corrected input changed the result. Give the run a particular table version so we can tell a new model from a new dataset.
- 04Ready to exploreA Retried File Should Not Become New Training Data
A file finished, but its acknowledgement disappeared. Keep enough committed progress for the retry to recognise work that already reached the dataset.
- 05Ready to exploreThe Last Message to Arrive May Be an Older Reading
A newly arrived file can contain an old correction. Keep each pump’s source version, including its deletion, so delivery order cannot choose the current reading.
Publishing a pipeline result
A pipeline finds an invalid reading. Should it record the failure and continue, leave that row out, or stop the update? Those choices change what reaches the model.
- 06Ready to exploreA Quality Check Needs a Decision About Failure
The dashboard found an invalid reading, yet training still received it. Decide what a failed check should change in the published dataset.
Connecting model evidence to an artifact
One model run has a lower error than another. Before we prefer it, we need to know what each run was evaluated on. MLflow can keep those details alongside the model files.
- 07Ready to exploreA Better Score on Which Dataset?
A lower error is tempting. First find out which observations produced that score and which saved model belongs to it.
Making feature access explicit
A training job can name a table and still lack permission to read it. A feature can also be present now but unavailable at the time of an older prediction. We’ll handle those as separate requirements.
- 08Ready to exploreSELECT Is Not the Whole Permission to Read a Table
The table is there, and a colleague can read it. Why does the scheduled training job still fail to access the same records?
- 09Ready to exploreGive Each Prediction the Features It Could Actually Have Seen
The current temperature is useful for a prediction now. An old prediction needs the value that was available then, including a deliberate rule for missing or stale readings.