Keep Enough of a Calculation to Run It Again

Botie has prepared a feature table, fitted coefficients, and kept the greenhouse names. Tomorrow's program must load those pieces in the same order. Saving only a screenshot or a rounded list of predictions would not be enough to repeat the calculation.

NumPy can save one array in its native format, keep several arrays in a named archive, or exchange numbers as text. We will try each route and check both the values and the information needed to interpret them.

The feature table, coefficients and names have already been loaded. Follow one row through the calculation and check that its name still identifies the resulting score.

[
  [
    24,
    46
  ],
  [
    "north",
    "south"
  ]
]

np.save writes one array in NPY format. np.load reads it back, including the dtype and shape. A printed string is intended for people; a saved numerical array is intended for another calculation.

These browser examples use BytesIO, a binary file held in memory. It follows the same read/write interface as a file without writing to your computer. Its contents are temporary and disappear when the Python runtime is reset. For a local Python program, a file path can be supplied instead.

import numpy as np

from io import BytesIO

readings = np.array([[12.,18.],[14.,22.]], dtype=np.float64)
file = BytesIO()
np.save(file, readings, allow_pickle=False)
file.seek(0)
restored = np.load(file, allow_pickle=False)
print(restored.tolist())
print(restored.shape)
print(restored.dtype)
print(np.array_equal(restored, readings))

[[12.0, 18.0], [14.0, 22.0]]
(2, 2)
float64
True

An NPZ archive can hold several arrays. Keyword arguments to savez choose names by which they can be retrieved. This is useful when a feature table, a coefficient vector, and observation labels belong to the same calculation.

The archive is a resource that should be closed after reading. A with statement keeps it open for the indented block and closes it when the block ends. Keep any arrays needed later by assigning them within that block.

import numpy as np

from io import BytesIO

X = np.array([[10.,40.],[20.,60.]])
weights = np.array([2.,0.1])
names = np.array(["north","south"])
file = BytesIO()
np.savez(file, features=X, weights=weights, names=names)
file.seek(0)
with np.load(file, allow_pickle=False) as saved:
    restored_X = saved["features"]
    restored_weights = saved["weights"]
    restored_names = saved["names"]
print(restored_names.tolist())
print((restored_X @ restored_weights).tolist())

['north', 'south']
[24.0, 46.0]

Text is convenient for inspecting and sharing rows, but a reader must know the delimiter, which lines are headers, and what type to use. loadtxt is suitable for a regular numeric table with the stated format. genfromtxt provides additional handling for missing entries.

StringIO is the text counterpart to BytesIO. It lets this example read a tiny CSV entirely in memory. On disk, the reader can take a path instead.

import numpy as np

from io import StringIO

csv = StringIO("""temperature,humidity
12,50
18,60
""")
features = np.loadtxt(csv, delimiter=",", skiprows=1, ndmin=2)
print(features.tolist())
print(features.shape)

[[12.0, 50.0], [18.0, 60.0]]
(2, 2)

A round-trip check should compare the loaded shape, dtype, values, and label ordering with the originals. Units, feature meanings, random settings, and library versions belong in the experiment record too. A saved array cannot reconstruct assumptions that were never recorded.

When writing text with savetxt, the format controls precision. A small number of decimal places may be fine for a human report, but it deliberately discards information needed for an exact numerical replay.

import numpy as np

from io import StringIO

original = np.array([[12.345,50.],[18.765,60.]])
text_file = StringIO()
np.savetxt(text_file, original, delimiter=",", fmt="%.2f")
text_file.seek(0)
restored = np.loadtxt(text_file, delimiter=",", ndmin=2)
print(restored.shape == original.shape)
print(np.array_equal(restored, original))
print(np.allclose(restored, original, rtol=0, atol=0.01))

True
False
True

A file can load and still be the wrong version, have the wrong column order, or contain the wrong dtype. Compare a saved and loaded array's shape, dtype and values. Then check its meaning against the metadata that names its columns and units.

For numerical arrays, prefer formats that preserve these numerical properties directly. Avoid object arrays when plain numeric or string arrays express the data. Loading pickled objects requires trust in the file and is unnecessary for the measurements here.

import numpy as np
import io
original = np.array([[12., 50.], [18., 65.]], dtype=np.float64)
buffer = io.BytesIO()
np.save(buffer, original, allow_pickle=False)
buffer.seek(0)
restored = np.load(buffer, allow_pickle=False)
print(restored.shape == original.shape)
print(restored.dtype == original.dtype)
print(bool(np.array_equal(restored, original)))

True
True
True

Imagine opening this dataset in a fresh Python process a month from now. Would we still know which column is temperature, what its units are, and how a missing reading was recorded? Keep those details beside the arrays, along with the period the records came from. For a fitted transformation, save the training reference in its feature order too.

Give changes to the file layout a version that the loader can check. If a version is unsupported, report that explicitly. Guessing the meaning of an unfamiliar two-column file could give us convincing numbers for the wrong calculation.