Writing the Steps Is Different from Running Them

Suppose our pump readings have grown from one file we can inspect to years of files arriving from several sites. We still need to remove the readings our sensor contract calls invalid. Now we also need somewhere for the team to run that calculation and see which data it used. Databricks is a managed cloud platform for this shared data and machine-learning work. In these lessons, Spark is the computation engine we use inside that environment.

We will begin with a small puzzle from that larger job. Botie asks for the temperatures above a declared minimum, then looks at the original table. The broken reading is still there. Did the filter fail? Before running a bigger job, we need to understand what names the source records, what describes a proposed result and what actually asks Spark to calculate it.

The original table still contains rejected readings. Botie needs the result of the cleaning procedure.

[
  {
    "pump": "P7",
    "temperature": 12
  }
]

One file of pump readings is easy to inspect with a Python program. Years of readings from many sites are a different workload. The files may exceed one machine’s memory, or the team may need an answer sooner than one process can produce it. We can divide work into portions and give several processes the same declared operation.

Apache Spark is an engine for organising and executing that kind of calculation. A portion of the records is called a partition. For a filter, separate partitions can apply the same condition independently; a later grouped total may have to combine their contributions. Spark can also run locally for small work, so its name does not by itself mean that several machines are involved.

A driver coordinates the application. Executor processes carry out tasks on the available compute. Describing operations through Spark’s table interface lets the engine choose how to carry them out while preserving the requested result.

Adding workers changes where the calculation runs. We still have to explain which records the calculation should return.

A working filter leaves the team with practical questions. Where will the scheduled job run? How will a colleague inspect the code and the records from a failed attempt? Providing a shared environment is work of its own.

Databricks is a managed cloud platform for this work. Its workspace gives a team a place to develop and run data and machine-learning jobs while the service manages supported compute and connects the work to configured storage and access controls. In these lessons we use Spark there for the table calculations. Apache Spark also exists independently as an open-source project.

Choosing the platform does not choose a cleaning rule for us. We still decide what an invalid sensor value means and what a model may learn from it. Later lessons will follow the other responsibilities that appear once the calculation is shared: keeping a table version, recording a training attempt and checking the job’s permission to read an input.

The local workshops let us inspect those decisions on a handful of records. The optional code references show where the same ideas meet a real Databricks workspace.

A Python list can hold several dictionaries. Each dictionary keeps one pump identifier beside its temperature and status. That way, selecting or rejecting a reading keeps the fields describing that observation together. A column is one named field read across those records.

Spark’s DataFrame is a table-shaped object with named columns and declared data types. We can ask for a new DataFrame by describing which rows and columns we want. The original object continues to describe the source. Later, when we request rows, Spark’s driver process coordinates the calculation and executor processes carry out its tasks.

Follow the named records first. The small program below reads their pump identifiers while leaving each complete dictionary available to inspect.

records = [{"pump": "P7", "temperature": 12}, {"pump": "P2", "temperature": -3}]
print([row["pump"] for row in records])

['P7', 'P2']

Give the source a name, raw, and the requested result another name, clean. A transformation is an operation that describes a new result from an earlier one. A filter keeps the rows that satisfy a condition. A projection keeps the named columns we request.

For our sensor example the condition is temperature at least zero. We are checking that particular field. A different column describing a temperature change could legitimately be negative, so the rule needs both a field name and its meaning.

raw temperatures = [12, -3]
keep temperature >= 0
clean temperatures = [12]
raw temperatures remain [12, -3]

After the transformation, raw still names the source and clean names the requested subset. We can inspect either one deliberately.

Spark transformations are generally lazy: they construct a plan rather than immediately materialising every intermediate table. An action such as count, show or collect asks the engine for a result. Analysis and some metadata work can still happen before the action, so lazy does not mean absolutely nothing happens.

The browser model makes this distinction concrete. Plan a numerical filter and Plan the output columns return instructions. Execute the plan receives those instructions and a batch. Reading the plan alone is different from inspecting its output.

When timing execution, include the action and state which reading and transformations it requested.

The optional snippet builds a small DataFrame with an explicit schema: a declaration of column names and data types. The where expression returns another DataFrame. explain displays its plan; show requests displayed rows. Ordering here makes the small demonstration repeatable.

collect transfers the complete result to the driver process. That is useful for a genuinely small result, but defeats the purpose of distributed storage if the full training table cannot fit there. Keep large transformations on the workers and inspect bounded results.

A real physical plan may fuse operations or change their order while preserving their meaning. Our instruction list keeps the teaching order instead.

from pyspark.sql import functions as F

raw = spark.createDataFrame(
    [("P7", 12.0), ("P2", -3.0)],
    "pump string, temperature double",
)
clean = raw.where(F.col("temperature") >= 0).select("pump", "temperature")
clean.explain("formatted")
clean.orderBy("pump").show()

The next file may contain different pumps or more readings. A reusable cleaning procedure accepts the current records and applies the same declared rule. It should not return a result copied from the first demonstration.

In the workshop, the same plan runs against every supplied case. In Python, write a function whose arguments carry the records, threshold and requested columns. Keep their roles separate. A fresh list of result dictionaries also prevents a later edit of the result from changing the source records.

Where should matching pump contributions meet when they begin on several workers?