A Quality Check Needs a Decision About Failure
A Pipeline Needed More Than a Working Query
BOTIE
Databricks announced Delta Live Tables in May 2021 to let authors declare datasets, their dependencies and quality expectations together. The pipeline system could inspect those definitions and manage their updates. The current interface used here is Lakeflow Spark Declarative Pipelines. Delta Lake remains the versioned storage underneath; a pipeline is the process that builds and refreshes datasets.
An expectation is a named condition that a row should satisfy. It still needs a policy: keep and report the failing row, exclude it, or stop this update. We will keep the condition and its consequence visible as separate components, then inspect the actual rows training receives.
Sources for this history
The Dashboard Counted Bad Readings, but Training Still Used Them
Botie’s pipeline reports that one temperature is invalid. The next model still trains on it. The check ran; it simply did not have the effect the team assumed. Did we intend to record a warning, exclude the row or prevent this dataset update?
We need to separate the condition that marks a row as invalid from the policy applied when that condition fails. A visible failure count is evidence. Whether the output may be published is another decision.
We evaluate one candidate dataset and one temperature condition, then apply its publication policy. A real Lakeflow pipeline has its own update boundaries and failure metrics; a rejected-record dataset must be designed explicitly.
Step 1 of 1
The predicate and the policy are separate
The declared valid-temperature range in this fixture begins at zero.
| Row | temperature | Predicate passes |
|---|---|---|
| r1 | 10 | true |
| r2 | -1 | false |
| r3 | 20 | true |
r1
- temperature
- 10
- Predicate passes
- true
r2
- temperature
- -1
- Predicate passes
- false
r3
- temperature
- 20
- Predicate passes
- true
State what a passing row means
The predicate produces evidence and the policy determines which output is acceptable.
Give a failed expectation an explicit effect
The quality dashboard counts invalid rows, but the model still receives them.
BOTIE
Publication status, rows and teaching check counts
{
"status": "failed",
"rows": [
{
"temperature": 4
}
],
"passed": 2,
"failed": 1
}Follow the operations
1. Candidate sensor records
[ { "temperature": 10 }, { "temperature": -1 }, { "temperature": 20 } ]2. Field checked by this expectation
temperature
3. Minimum accepted value
0
4. Evaluate a row expectation
[ { "row": { "temperature": 10 }, "passed": true }, { "row": { "temperature": -1 }, "passed": false }, { "row": { "temperature": 20 }, "passed": true } ]5. Declared publication policy
fail
6. Earlier published rows
[ { "temperature": 4 } ]7. Apply warn, drop or fail policy
{ "status": "failed", "rows": [ { "temperature": 4 } ], "passed": 2, "failed": 1 }8. Result
{ "status": "failed", "rows": [ { "temperature": 4 } ], "passed": 2, "failed": 1 }
We evaluate one candidate dataset and one temperature condition, then apply its publication policy. A real Lakeflow pipeline has its own update boundaries and failure metrics; a rejected-record dataset must be designed explicitly.
1. State what a passing row means
Suppose our sensor contract requires a temperature to be present and at least zero. We can test that rule on every row, but a boolean by itself tells the next part of the pipeline very little. Keep it attached to the reading that produced it.
Lakeflow calls a named quality condition an expectation. Its name identifies the check in reports; its condition supplies the rule to evaluate. Our local component returns each complete row beside a passed flag. It leaves the publication decision to the next component.
The lower bound belongs to this invented sensor contract. Other temperature measurements can legitimately be negative.
Keep each condition result beside its record ID
The boolean marks the condition only. It does not decide which rows should be published.
records = [{"id":"r1","temperature":10},{"id":"r2","temperature":None}]
checks = [(row["id"], row["temperature"] is not None and row["temperature"] >= 0) for row in records]
print(checks)[('r1', True), ('r2', False)]Warn, drop and fail consume this evidence differently. The later challenge implements those consequences.
2. Choose what the failed condition should do
A warn policy retains the candidate records while recording the condition. A drop policy excludes failing rows from that output. A fail policy stops this dataset update when its condition fails. These policies answer different operational questions.
Dropping a rare fault case can bias the dataset. Stopping every update for an optional descriptive field can unnecessarily block useful work. Choose a policy from the meaning and consequences of the field, and preserve the evidence needed to inspect failures.
Choose the failure consequence from what downstream training is allowed to use.
3. A failed update is not a new successful snapshot
In our atomic candidate-publication model, fail preserves the earlier published rows when any candidate row fails. The result also identifies that this attempt failed. Do not relabel the older dataset as newly validated output.
This is a boundary around one teaching dataset. Real pipelines can contain independent flows and datasets; failing one update is not a general promise that every separate effect in the pipeline has been rolled back. Platform metrics for a failed update can also differ from the counts we calculate locally here.
candidate readings = [10, -1, 20]
passing rows = 2
failing rows = 1
policy = fail
published rows remain the earlier snapshotHere the counts describe the candidate, while the returned rows identify which snapshot remains available.
Check yourself
Questions on Parts 1 to 3
Question 1 of 2
Which rows does this declared policy publish?
Use the information below.
Candidate temperatures [10,-1,20], minimum zero, policy warn.4. Attach an expectation to a Lakeflow dataset
A materialized view stores the result of a query, which the pipeline can refresh from its declared sources. The reference describes one and attaches an expectation that fails its update. It belongs in a Lakeflow pipeline source file with a configured catalog and schema. The function returns a DataFrame; the pipeline manages its evaluation.
Use the current pyspark.pipelines interface in a supported Databricks runtime. Expectations shown here are Databricks extensions, not a claim that every open-source Spark installation supplies the same decorators. Keep arbitrary side effects out of dataset definitions.
Use expect to retain rows and record checks, or expect_or_drop to remove invalid records from that output, when those policies are appropriate. They are different from expect_or_fail.
Run in your own environment
Fail an invalid candidate dataset in Lakeflow
This optional reference runs in your own Databricks environment. The browser exercise below uses ordinary Python and does not contact Databricks.
Add this source to an optional Databricks Lakeflow pipeline configured with an approved target catalog/schema and an existing raw source table. Running the pipeline materialises data and can incur charges.
Use a runtime providing the documented pyspark.pipelines and Databricks expectation APIs. This is not an ordinary standalone Pyodide script.
from pyspark import pipelines as dp
@dp.materialized_view(name="validated_sensor_readings")
@dp.expect_or_fail(
"temperature_is_usable",
"temperature IS NOT NULL AND temperature >= 0",
)
def validated_sensor_readings():
return spark.read.table("lab.training.raw_readings")A failed expectation blocks this dataset update. Test the actual pipeline’s dependency and failure behavior separately from the local policy simulation.
Official interface documentation. Documentation checked .
5. Quarantine is an output you must design
Dropping a row from one result does not automatically create a quarantine dataset. If the team needs to inspect rejected readings, preserve them in an approved raw or rejected-record path with their failure reason and input identity. That path still needs access and retention rules.
Keep the source immutable while fixing the transformation or producer. A corrected input should produce a new explainable result, rather than silently rewriting the evidence behind an old model.
The earlier quarantine lesson supplies the row-level reasoning. Here we make sure a platform policy does what that reasoning requires.
6. Test the output, not only the check name
Give the same candidate batch to warn, drop and fail policies. Compare rows as well as counts and status. Then use a batch where every row passes and check that fail does not prevent an acceptable update.
A fresh missing value tests another branch of the predicate. Keep these tests separate from a real pipeline dry run or integration test: our Python function can establish the local policy logic without establishing cluster permissions or storage atomicity.
The next lesson connects a model’s measured score to the dataset and split on which that score was obtained.
Check yourself
Questions on Parts 1 to 6
Question 1 of 4
That alone automatically creates a rejected-record table with a retention policy.
Consider this example.
expect_or_drop removes an invalid row from its declared result.Try it with Botie
Build with Botie
Write and run Python