Data Engineering
A model learning from pump readings needs a dependable record of what each pump was doing. Work from incoming files to a training table, then examine how Databricks handles those records as their volume grows.
Data Pipelines for Learning
A training table is the result of decisions about records. Follow Botie from unequal input files to historical features, rejected sensor batches and late events. Keep enough history to explain which data actually produced a model.5 lessonsDatabricks for ML Pipelines
Databricks gives a team a shared environment for processing data and building ML applications. Apache Spark is one of the tools that performs the calculations, with work spread across several machines. We’ll start with why that became useful, then examine what a training run needs to remember about its data and model files.9 lessons