Which Version of the Table Did the Model Read?
How Could Readers Agree While Files Were Changing?
BOTIE
Delta Lake puts a transaction log alongside the data files. A successful commit records which files form the next table version. A reader can use one committed version even while later work creates another. Databricks released Delta Lake as an open-source project on April 24, 2019; the 2020 research paper explained the design for reliable analytical tables over object storage.
Spark performs the table calculation. Delta Lake supplies the versioned table storage that this read relies on. On the floor we keep full local snapshots so you can see exactly which records a version selects. The real format uses its log and retained files to reconstruct that state.
The Same Training Script Now Reports a Different Mean
Botie saves the cleaning code and runs training again. The code has not changed, but a corrected sensor file has. The rerun reads today’s table while yesterday’s model read an older one. Comparing their scores no longer isolates the code change.
A table name tells us where to look. It does not, by itself, identify which committed contents we used. We need a version that a run can name and read consistently.
The history contains full local copies of the supplied versions. A real Delta read also depends on its transaction log, retained files and concurrency rules; choosing versions of separate tables requires an explicit input contract.
Step 1 of 1
The correction belongs to another commit
The pump IDs are the same, but one reading changes.
| Snapshot | P7 temperature | P2 temperature |
|---|---|---|
| version 0 | 10 | 20 |
| version 1 | 30 | 20 |
version 0
- P7 temperature
- 10
- P2 temperature
- 20
version 1
- P7 temperature
- 30
- P2 temperature
- 20
Separate a table address from its contents
A version choice occurs before numerical extraction, so the calculation cannot accidentally mix committed states.
Keep the run on its declared table version
The same procedure changed its answer after a source correction.
BOTIE
Mean from the declared committed version
15
Follow the operations
1. Retained committed snapshots
[ { "version": 0, "rows": [ { "pump": "P7", "temperature": 10 }, { "pump": "P2", "temperature": 20 } ] } ]2. Version named by this run
0
3. Open a committed table version
[ { "pump": "P7", "temperature": 10 }, { "pump": "P2", "temperature": 20 } ]4. Numerical feature column
temperature
5. Read a named numerical column
- 010
- 120
6. Mean or an explicit missing result
15
7. Result
15
The history contains full local copies of the supplied versions. A real Delta read also depends on its transaction log, retained files and concurrency rules; choosing versions of separate tables requires an explicit input contract.
1. Separate a table address from its contents
The name lab.training.sensor_features tells us which table to open. The name alone does not freeze what is inside it. Suppose someone corrects a reading tomorrow. The address can stay the same while the records behind it change.
A version identifies one committed state of that particular table. Keep the table identity beside its version when recording a model’s inputs. In our Python example, history is a list of dictionaries, each holding a version and its complete rows. Those outer versions belong to table states; they are different from the per-pump changes we will meet later.
The useful record is specific: this model read this table at this version.
Inspect the nesting of a retained snapshot
The outer list holds table versions. Inside one matching dictionary, rows holds the complete sensor records.
history = [{"version":0,"rows":[{"pump":"A","temperature":10}]}, {"version":1,"rows":[{"pump":"A","temperature":30}]}]
matching = [entry for entry in history if entry["version"] == 0]
print(matching[0]["rows"])[{'pump': 'A', 'temperature': 10}]The challenge must additionally reject absent or duplicated version identities and handle an empty committed snapshot.
2. Read one committed state for this calculation
Select the requested version before extracting the temperature column. A Delta commit records one complete table update as a published version; the mere presence of some new data files does not make them the current table. All readings in the calculation then come from that supplied snapshot. The correction in a later commit does not replace a value inside an already selected older snapshot.
A committed snapshot is the boundary we rely on here. It is not permission to combine unrelated versions of several tables and call the result a single transaction. Record the required version of every independently managed input.
version 0 temperatures = [10, 20]
mean = (10 + 20) / 2 = 15
version 1 temperatures = [30, 20]
mean = (30 + 20) / 2 = 25The changed mean follows the corrected input. A comparison meant to isolate a code change must keep that input fixed.
Check yourself
Questions on Parts 1 to 2
Question 1 of 2
Which values belong in its calculation?
Use the information below.
version 0 contains temperatures [10,20]. version 1 contains [30,20]. A run declares version 0.3. Pin a Delta read explicitly
The reference asks Delta for a specific retained version. Another writer can commit a newer table state without turning this read into a request for latest. Save the chosen version with the model’s training metadata.
A current table read is useful when we intentionally want current data. A reproducibility claim needs a declared input identity. Avoid querying latest in one part of a workflow and assuming another query still means that same state.
Check that the requested table version is retained before promising a future rerun.
Run in your own environment
Open an explicit Delta version
This optional reference runs in your own Databricks environment. The browser exercise below uses ordinary Python and does not contact Databricks.
Use a Delta table in an optional Databricks workspace, replacing the example name and version with a table and retained commit you may read.
This reference reads only. It does not create an archive or change retention settings.
table_name = "lab.training.sensor_features"
training_version = 0
training = (spark.read.format("delta")
.option("versionAsOf", training_version)
.table(table_name))
training.select("pump", "temperature").orderBy("pump").show()Record the table identity and version, and distinguish them from the model version. A later commit should not alter this declared read.
Official interface documentation. Documentation checked .
4. A retained version is not an indefinite backup
Older snapshots need both sufficient log history and the data files they reference. Retention and cleanup can make an old version unreadable. A version number written in a report does not recreate deleted files.
Plan retention around reproducibility and recovery requirements. Export or otherwise preserve approved training inputs when that requirement exceeds ordinary table retention. Privacy and deletion requirements can also constrain what may be kept; do not copy personal data indefinitely merely to make a model reproducible.
Our Python function raises ValueError when the requested version is absent. Returning the latest version would silently answer a different question.
5. Time travel and historical features answer different questions
Delta time travel asks which committed state of a table to read. A historical feature lookup asks which feature value belongs to a particular prediction time inside that table. One table snapshot can contain feature rows for many prediction times.
We will use both ideas later. Pinning a snapshot does not automatically remove future features from a join, and filtering feature timestamps does not make an overwritten table version recoverable.
The next ingestion lesson will make a retry safe without appending the same discovered file again.
Check yourself
Questions on Parts 1 to 5
Question 1 of 4
What should a strict reproducibility read do?
Use the information below.
A required old version is absent because its needed files were removed.Try it with Botie
Build with Botie
Write and run Python