Who Checks the Model-Choosing Process?

Botie tries several feature sets and regularisation strengths. One has the lowest validation error, so we keep it. That is a reasonable development decision. Reporting that same winning score as an independent assessment is another matter.

The validation results helped choose the winner. Some of its advantage may belong to the particular observations we happened to use. We need fresh observations outside that choosing process.

Select each fold’s candidate using its inner validation losses. Only then read that candidate’s outer loss.

4.5

A frozen candidate assessed on a held-out test set answers a question about that fitted candidate. Nested cross-validation assesses a procedure that includes selecting settings and fitting. Different outer folds may choose different settings.

Neither description makes the other unnecessary. A development team may use nested evaluation to compare procedures, then train its final chosen pipeline and assess it on a separately reserved dataset.

Write down the object being evaluated before selecting a split design.

Set aside one outer test portion. The rest is the outer training portion, containing every record available to the inner search. The inner search divides that portion again to fit candidate pipelines and score them.

The outer test labels cannot select features, thresholds, stopping epochs or hyperparameters. Treat all those choices as part of the procedure. A manual feature decision informed by outer errors is still selection.

Outer training → inner training + inner validation
Outer test → assessment only

For new patients or machines, split at that group boundary. For forecasting, preserve the available history at each origin. Nesting does not fix an inappropriate split.

Average each candidate’s inner validation losses using a declared aggregation rule. Select the candidate with the lowest loss. Refit its complete pipeline on the outer training portion, then predict the outer test portion once.

Repeat across outer folds. Report the outer assessment results, not the average of the best inner scores. Declare whether folds receive equal weight or predictions are pooled; those differ when fold sizes differ.

Inner choices: [A, B]
Selected outer losses: [4, 5]
Equal-fold mean = (4 + 5) / 2 = 4.5

Our teaching tables assume equal-size outer folds. The practice function explicitly reports an equal-fold mean.

Imputers, scalers, feature selectors and learned encoders must be fitted on the current training partition. Even an unsupervised scaler can leak distribution information if it sees the outer test data before evaluation.

In scikit-learn, a Pipeline inside GridSearchCV keeps fitted preprocessing with the estimator. Cross-validating that search object creates the outer assessment. Supplying a transformed full dataset before the search defeats that boundary.

outer training
  → fit inner pipeline on each inner training split
  → score each inner validation split
  → select settings
  → refit entire pipeline on outer training
  → transform and predict outer test

For grouped searches, ensure both levels actually receive and respect group information. API convenience does not establish a correct information boundary.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(7)
X = rng.normal(size=(60, 2))
y = 10 + 2 * X[:, 0] + 4 * X[:, 1] + rng.normal(size=60)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("ridge", Ridge()),
])
inner = KFold(n_splits=3, shuffle=True, random_state=11)
outer = KFold(n_splits=5, shuffle=True, random_state=19)
search = GridSearchCV(
    pipeline, {"ridge__alpha": [0.1, 1.0, 10.0]},
    scoring="neg_mean_squared_error", cv=inner, refit=True,
)
outer_mse = -cross_val_score(
    search, X, y, cv=outer, scoring="neg_mean_squared_error"
)
print("Outer MSEs:", outer_mse)
print("Equal-fold mean MSE:", outer_mse.mean())

# After settling the procedure, fit it on all development observations.
# This fitted object's best_score_ is an inner selection score.
search.fit(X, y)
print("Final development settings:", search.best_params_)

The outer folds are assessment runs, not contestants. Picking the fold with the best score creates another selection. After settling the procedure, run its search on the full development dataset and refit the selected pipeline there.

If you change the search space after inspecting outer results, those results have informed development. Reserve new evidence for the resulting procedure. Report the number and nature of those decisions.

Fold scores share overlapping training data. Their standard deviation is descriptive and is not automatically a valid standard error from independent experiments.

In the tables below, axis 0 is outer fold and axis 1 is candidate. argmin along axis 1 gives one choice per fold. Advanced indexing then picks one outer score from each corresponding row.

The challenge combines the two operations and checks that malformed tables cannot silently broadcast.

import numpy as np
outer = np.array([[8., 3.], [4., 7.]])
choices = np.array([1, 0])
print(outer[np.arange(2), choices].tolist())

[3.0, 4.0]

Nested evaluation repeats fitting many times, so it can be expensive. A sufficiently large independent test set with a disciplined development process may answer the immediate question more simply. If data are limited and model selection is substantial, nesting helps expose that selection to independent outer cases.

Record the candidate space, splits, seeds and software versions. A repeated experiment should reproduce the protocol, not merely a rounded final average.

Use nesting where it answers the evaluation question. More folds cannot manufacture missing populations or future conditions.