A Perfect Fit Left Two Different Answers

A Good Fit Could Still Leave the Parameters Unknown

BOTIE

Suppose every calibration test turns two heaters up together. A fitting program can learn their combined heat output while leaving each heater’s contribution unresolved. Repeating that same operating pattern gives it more records of the combined effect, without asking the question that separates the gains.

A least-squares fit chooses coefficients by minimising the sum of squared prediction errors. Computational work on these fits had to address both noisy equations and weakly determined answers. In 1965, Gene Golub and William Kahan described a numerically stable way to calculate singular values and a matrix’s pseudo-inverse, which extends the idea of an inverse to such fitting problems. These calculations can expose changes to the coefficients that the measurements constrain strongly, weakly or not at all.

That historical numerical work does not tell us which heater is stronger. It helps us recognise when a returned coefficient is a convention rather than an independently measured effect. We will first design a setting on which two fitting models disagree, then give the missing information its linear-algebra names.

Sources for this history

Which Heater Actually Produced the Heat?

Botie is fitting a small physical model to produce simulator training data. Two heaters always ran at the same setting during the recorded experiments. The measured heat is easy to predict, but several different pairs of heater gains fit every observation. Which pair belongs in the simulator?

The fitting program can return a pair of numbers without the measurements determining that pair. We will first see what the observations actually constrain. Then we will design a measurement that can tell two plausible fitted models apart.

The two candidate gain vectors and additive heater model are supplied. The gap threshold helps plan a distinguishing measurement; the actual equipment, uncertainty and permission to run that experiment still need assessment.

Step 1 of 1

Both models explain the experiments we already ran

The settings are recorded in a common unit. Gains convert setting into heat under this simplified additive model.

The final row is a proposed new experiment. It distinguishes these two candidates; it does not establish which candidate is correct.
Experiment settings	Model A gains [1,2]	Model B gains [2,1]
Both at one: [1,1]	3	3
Both at two: [2,2]	6	6
Only heater one: [1,0]	1	2

Both at one: [1,1]
Model A gains [1,2]
3
Model B gains [2,1]
3
Both at two: [2,2]
Model A gains [1,2]
6
Model B gains [2,1]
6
Only heater one: [1,0]
Model A gains [1,2]
1
Model B gains [2,1]
2

The final row is a proposed new experiment. It distinguishes these two candidates; it does not establish which candidate is correct.

Look at what varied in the original experiments

New evidence must distinguish the competing explanations, rather than repeat their shared prediction.

Ask the heaters a question that separates the fits

Botie’s calibration fit predicts the recorded totals, but two different gain vectors explain them equally well.

BOTIE

Botie’s calibration fit predicts the recorded totals, but two different gain vectors explain them equally well.

Choose an example

Proposed heater-setting rows

1

2

1

2

3

1

2

Candidate gain vector A

1

2

Candidate gain vector B

1

2

Minimum planning gap in heat units

Experiment positions that separate the candidate predictions

01
12

Follow the operations

1. Proposed heater-setting rows
1 1
1 0
0 1
2. Candidate gain vector A
1. 01
2. 12
3. Matrix times vector
1. 03
2. 11
3. 22
4. Candidate gain vector B
1. 02
2. 11
5. Matrix times vector
1. 03
2. 12
3. 21
6. Subtract
1. 00
2. 1-1
3. 21
7. Absolute value
1. 00
2. 11
3. 21
8. Minimum planning gap in heat units
```
0.5
```
9. Threshold gate
1. 00
2. 11
3. 21
10. Keep the passing experiment positions
1. 01
2. 12
11. Result
1. 01
2. 12

The two candidate gain vectors and additive heater model are supplied. The gap threshold helps plan a distinguishing measurement; the actual equipment, uncertainty and permission to run that experiment still need assessment.

1. Look at what varied in the original experiments

Each row holds the two heater settings for one experiment. The response array holds measured heat in the same row order. If the settings always rise together, the records show their combined effect. They do not show how much each heater contributed independently.

This can also happen in ordinary ML features. Two recorded inputs may repeat nearly the same information. A fitting routine still produces coefficients, but interpreting each coefficient as a separately measured effect requires care.

A shape check can confirm that arrays line up. It cannot establish that the experiments contain independent information.

2. Write the combinations that the data constrain

Our model adds the two setting-times-gain contributions. In the recorded experiments, each setting pair is a multiple of the same row. Changing one gain while compensating with the other leaves every fitted prediction unchanged.

There is a whole family of fitting answers here. We can predict the old operating pattern without having identified the individual gains.

prediction = setting_one × gain_one + setting_two × gain_two

[1,1] constrains gain_one + gain_two = 3
[2,2] constrains 2 × (gain_one + gain_two) = 6

[1,2], [2,1] and [1.5,1.5] all satisfy the observed total

Those coefficient pairs are alternative gain estimates, not additional rows of observations.

3. Choose a question on which the fitted models disagree

Evaluate both candidates at each proposed setting pair. Subtract their predicted responses and keep the magnitude. A large gap offers a way to distinguish these candidates if that experiment is feasible and the instrument can resolve the difference.

Running both heaters together again gives the same answer under both candidates. Running only one heater asks a different question. We are designing evidence, rather than simply collecting a larger file.

candidate settings = [[1,1], [1,0], [0,1]]
model A predictions = [3,1,2]
model B predictions = [3,2,1]
absolute gaps = [0,1,1]

The arithmetic ranks disagreement between the supplied candidates. Neither candidate is assumed to be the real physical system.

4. State the measurement resolution before declaring a distinction

A prediction gap can be too small to distinguish with the available instrument. The workshop uses a declared positive minimum gap, in response units, and keeps every candidate experiment reaching it. This is a simple planning rule, not a statistical test.

Actual experimental uncertainty may include systematic bias, correlated errors and changing equipment conditions. A planned gap above this threshold does not guarantee that one noisy observation will identify the correct model.

minimum planning gap = 0.5 heat units
gaps = [0,1,1]
passing original positions = [1,2]

An empty result means these candidates cannot be separated by the proposed settings at the stated resolution. Do not replace it with an arbitrary experiment.

Check yourself

Questions on Parts 1 to 4

Question 1 of 2

Which candidate fits both recorded responses?

Use the information below.

Observed settings [[1,1],[2,2]], responses [3,6]. Candidate gains are [1,2] and [2,1].

5. Now name the missing information

An identifiable parameter is one the allowed observations can distinguish from other parameter choices under the stated model. Our old heater tests fail that requirement: we can increase one gain and reduce the other without changing any predicted total for those settings.

Linear algebra describes the missing distinction as a direction in parameter space that these measurements cannot reveal. Rank counts the independent directions represented by the measurement matrix. Full column rank removes this particular linear ambiguity; we still need evidence that the additive heater relationship describes the equipment.

More repeated readings can reduce some measurement noise while leaving that same unresolved gain difference.

6. Almost the same experiment can still leave a fragile answer

The columns need not be exactly dependent to cause trouble. If they differ only slightly, a small measurement change can require a large coefficient change to maintain the fit. That sensitivity is a conditioning problem. That sensitivity comes from the problem and its representation. A different solver still has to work with the same measurements.

A matrix condition number quantifies worst-direction relative amplification under a stated norm. Its meaning depends on scaling and the mathematical problem. It is not a universal uncertainty interval for the fitted physical parameters.

well-separated rows: [[1,0], [0,1]]
nearly repeated rows: [[1,1], [1,1.000001]]

Both examples have full rank in exact arithmetic. The second provides much weaker separation of the two gains.

7. Inspect what the numerical fitting routine actually returned

NumPy’s lstsq returns more than a coefficient vector. It also reports numerical rank and singular values, which describe the strength of the matrix’s independent directions. A cutoff tells the routine which very small directions to treat as numerically unresolved.

When several coefficient vectors fit equally well, the routine chooses the one with the smallest Euclidean length. That choice is called the minimum-norm solution. It is a useful numerical convention, but it is not a measurement showing that two physical gains are equal.

Calculate prediction errors from the original rows as well. An empty residual array returned by the routine has a specific API meaning and cannot stand in for a complete diagnosis of the fit.

A fitting report should preserve how much information the measurements supplied, as well as how closely the chosen coefficients reproduce them.

See the unresolved direction in the fit report

This inspection prints numerical rank and the fitted predictions, rather than treating a coefficient vector as a direct measurement of each heater.

Python

import numpy as np
A = np.array([[1.,1.],[2.,2.]])
y = np.array([3.,6.])
fit,_,rank,singular = np.linalg.lstsq(A,y,rcond=None)
print(rank)
print(np.round(A @ fit,6).tolist())

Output

1
[3.0, 6.0]

The recorded totals are fitted while only one independent parameter direction is numerically resolved.

8. Take a new measurement and be willing to reject both models

Keep the proposed experiment separate from the measurement eventually obtained. If the response disagrees with both candidates, inspect the model structure, calibration and experimental conditions instead of forcing the result into one of the old explanations.

The same discipline matters when a learned simulator becomes training data for an agent. Preserve which operating conditions identified its parameters and test conditions where that evidence was weak.

A good training fit, an identified parameter and a model that transfers are different claims. Keep the evidence for each one visible.