Would More Examples Help?

Before collecting more data or choosing a more flexible model, we can examine how training and validation errors change as the training set grows.

13 sections · 3 questions · 2 coding challenges

1 / 13

The arrow keys move between sections too.

Model Comparisons Depend on How Much Data We Have

BOTIE

A comparison at one dataset size can hide how two learning methods respond as more examples become available. A method that works well with few observations need not remain the best choice with many.

Perlich, Provost, and Simonoff's 2003 study compared tree induction and logistic regression through learning curves. Our supplied greenhouse errors are invented illustrations, not results from that study or a model trained in this widget. They let us inspect the interpretation before running larger fitting experiments.

Sources for this history

Should We Collect More Readings or Change the Rule?

Botie's temperature model makes errors on new days. We could collect more days, or choose a more flexible rule. Those changes address different problems, so it helps to inspect the behaviour we already have.

We can investigate by fitting the same procedure with different amounts of training data. At each size, measure how well it fits those observations and how well it predicts a separate validation set. Plotting the two errors gives us learning curves. Their shapes can help us decide what to try next, although another batch of observations may behave differently.

These curves are illustrative measurements for reasoning about training behaviour.

Step 1 of 3

Fit with different numbers of examples

Each column refers to the same model family fitted with a different amount of training data.

Training examples

Training run 110
Training run 250
Training run 3100

Step 2 of 3

Keep the two errors separate

Training error measures the fitted examples. Validation error measures examples excluded from fitting. These illustrative curves show the two quantities coming closer as the training set grows.

Vertical: Mean squared error

Horizontal: Number of training examples

Invented errors for illustration. The lower green line is training MSE; the upper red line is validation MSE. No model was fitted to produce these teaching values.

Step 3 of 3

Watch the gap alongside the actual errors

The gap narrows here as more data arrive. A narrow gap alone would not be enough if both errors remained large.

An invented example for this lesson.
Training set size	Validation minus training
10 examples	8
50 examples	4
100 examples	1

10 examples
Validation minus training
8
50 examples
Validation minus training
4
100 examples
Validation minus training
1

The gap and the error level answer different questions

Each subtraction compares the same training size. Read these gaps beside the error levels before deciding what to try next.

Are the errors improving, or merely getting closer?

A small gap between training and validation can still mean both errors are poor. Compare the gap with the actual error levels as the supplied training size changes.

BOTIE

Botie has training and validation errors for matching training sizes. We need to see their differences without confusing a narrow gap with low error.

Choose an example

Illustrative training errors

Illustrative validation errors

Validation error minus training error

Follow the operations

1. Illustrative validation errors
1. 09
2. 16
3. 24
2. Illustrative training errors
1. 01
2. 12
3. 23
3. Subtract
1. 08
2. 14
3. 21
4. Result
1. 08
2. 14
3. 21

These errors are supplied teaching illustrations. No model is fitted by this construction.

1. Change the Training Size, Not Everything at Once

Choose several training sizes. At each size, fit the same procedure from scratch, including preprocessing. Evaluate the fitted model on its training subset and on data excluded from that fit.

Keep the metric and validation design consistent. Repeat subsets or splits when possible so an unusually easy selection does not masquerade as a trend.

A learning curve varies the amount of training data. A validation curve instead varies a model setting. The axes answer different questions.

2. Inspect Both Curves Before Summarising Them

Low training error with much larger validation error suggests the fitted rule depends heavily on details of the training sample. If the gap shrinks as data increase, collecting more comparable data may help.

If both errors stay high and close together, the current representation or model may be too limited, although noise, poor measurements, or distribution mismatch can produce similar symptoms.

At 10 examples: gap = 9 - 1 = 8
At 50 examples: gap = 6 - 2 = 4
At 100 examples: gap = 4 - 3 = 1

The supplied illustration shows a narrowing gap and improving validation error. It is not a universal shape that all models must follow.

3. Do Not Mistake a Small Gap for a Good Predictor

Subtracting the two errors measures their difference, not their absolute quality. Equal errors can both be unacceptable. Compare the validation score with the baseline and with the needs of the application.

Training MSE = 20
Validation MSE = 20
Gap = 20 - 20 = 0

This gap is smaller than in the earlier illustration, yet the validation error is much worse. A single gap score cannot replace the two curves.

4. Use the Pattern to Choose the Next Experiment

A curve can suggest whether to investigate more data, better features, regularisation, or a different model family. Test that suggestion rather than treating the visual pattern as a diagnosis with certainty.

The workshop calculates gaps from supplied errors. In a real study, obtaining each error requires a complete fit and evaluation at that training size. We keep those measurements explicitly separate from the arithmetic that summarises them.

Final test data should not guide repeated decisions about which model or dataset size to choose. Use validation data for those development decisions.

5. More Training Rows and More Training Passes Are Different Experiments

A learning curve against dataset size asks whether additional observations help. A curve against training steps asks what happens when the optimiser keeps revisiting the available data. Similar-looking plots can therefore answer different questions.

For the sample-size experiment, refit the complete procedure at each size and retain a fixed validation population. Repeat the subset selection when feasible. One unusually easy small subset can otherwise make the curve look much more convincing than the evidence supports.

6. Read the Pattern Before Choosing the Next Expense

High training and validation errors can mean the model cannot express the relationship, but also that optimisation failed or features are missing. Low training error with much higher validation error suggests a generalisation problem; check leakage, duplicates and group structure before calling it ordinary overfitting.

If validation performance is still improving with more representative data, collecting more may help. If it plateaus, investigate the errors rather than assuming a larger network is the answer. The curve narrows the investigation. It does not diagnose the cause on its own.

7. Report the Variation Behind the Average

Keep one result per repeat and sample size. Summarise across repeats only after checking that every column refers to the same size and every score uses the same validation procedure.

Three repeats at three training sizes

Rows are independently chosen training subsets; columns represent increasing dataset sizes. These supplied scores illustrate reporting rather than model fitting.

Python

import numpy as np
validation_error = np.array([[5., 4., 3.], [7., 5., 3.5], [6., 3., 4.]])
print(np.round(validation_error.mean(axis=0), 2).tolist())
print(np.round(validation_error.min(axis=0), 2).tolist())
print(np.round(validation_error.max(axis=0), 2).tolist())

Output

[6.0, 4.0, 3.5]
[5.0, 3.0, 3.0]
[7.0, 5.0, 4.0]

The mean improves, but the final size is not equally successful in every repeat. This range describes these runs, not a formal confidence interval.