A Small Covariance Can Be Confidently Wrong

Botie reports a narrow position estimate, but the simulated conveyor accelerated more than its motion model expected. The camera keeps disagreeing in the same direction. A smooth filtered line can hide that lag.

Is the camera unreliable, or is our prediction missing something about the movement? We need to inspect those assumptions and compare the result with a reference trajectory. Reducing the reported variance would only change what the filter claims about its uncertainty. It would not move an incorrect estimate to the right position.

The simulated part moves farther than Botie expected, yet the filter reports a narrow uncertainty. Compare the completed estimate with a separate reference to see that mismatch.

{
  "mean": 1.1,
  "variance": 0.1,
  "error": 0.1,
  "normalisedSquaredError": 0.1,
  "withinTwoSigma": true
}

The filter calculates covariance from the model and parameters we give it. To measure actual error, we need a separate reference position to compare with the estimate. Those are different sources of information. Unmodelled acceleration or a camera bias can leave the mean far from the reference even when the calculated covariance is small.

In the simulation we know the reference trajectory. With hardware, obtaining a trustworthy reference and accounting for its own uncertainty is an additional measurement problem.

Do not put reference truth into the inference branch to make the result look better. It belongs in evaluation.

A part can move differently from the motion prediction, and a camera can report a position differently from the actual one. We describe the movement error added in an interval with Q. R describes the observation error. Unexpected acceleration and a consistently shifted detector therefore give us different places to investigate.

Increasing Q usually gives an observation more influence in our scalar model. Increasing R usually gives it less. Neither operation proves that its new uncertainty is calibrated.

Persistent signed innovations suggest something systematic, but innovations alone do not uniquely identify which model is wrong.

The model expects a movement of 1 m. In the simulation, the part reaches 3 m and the camera also reports 3 m. A very small predicted variance tells the filter to trust its movement prediction strongly, so it only partly follows this useful camera correction.

earlier mean = 0 m; P = 0.1 m²
expected displacement = 1 m; Q = 0.01 m²
measurement = 3 m; R = 0.1 m²

predicted mean = 1 m; P⁻ = 0.11 m²
K = 0.11 / 0.21 ≈ 0.523810
corrected mean ≈ 2.047619 m
corrected variance ≈ 0.052381 m²

The remaining reference error is large relative to the declared spread. Inspect the motion assumption that produced that confidence.

A position error of a particular size is more surprising under a narrow uncertainty estimate than under a wide one. Divide the squared reference error by the declared variance to make that comparison. We also check whether the reference lies within two declared standard deviations of the estimated mean. These diagnostics assess the estimate; they do not choose a robot action.

With a correct scalar Gaussian error model, a two-standard-deviation interval has a known probability interpretation. A handful of hand-picked frames cannot establish that interpretation for a deployed detector and tracker.

error = estimate - reference
normalised squared error = error² / declared variance
inside two standard deviations when:
abs(error) <= 2 × sqrt(declared variance)

A very wide interval can cover the reference while providing little useful precision. Assess interval width and error together.

If the part has uncertain acceleration, that acceleration affects both its final velocity and how far it travels during an interval. Their errors are related. The matrix below describes one specific model: acceleration is a random constant over each interval, with a supplied variance. A continuously fluctuating acceleration model has a different process covariance, so the interpretation needs to accompany the matrix.

state = [position,velocity]
F = [[1,dt],[0,1]]
G = [[dt²/2],[dt]]
Q = G × acceleration_variance × Gᵀ

P⁻ = F P Fᵀ + Q

Acceleration variance here has units m²/s⁴ and refers to a random constant over that interval. It is not a continuous-time spectral density.

Two detectors using the same image may share errors. Applying both as independent observations can count information twice. A gate also selects which observations reach the filter, changing the distribution of the retained innovations.

A robot might plausibly be in either of two aisles. One Gaussian centred between them can put its central estimate in a place neither hypothesis supported. Nonlinear motion can create further difficulties. An extended Kalman filter approximates nonlinear relationships locally with linear ones; other filtering methods represent uncertainty differently. Each requires its own assumptions and evaluation.

An extended Kalman filter linearises nonlinear relationships; that approximation is not a guarantee that the full posterior remains Gaussian. Alternative filters also have assumptions and costs.

Use held-out motion patterns, gaps, detector failures and identity changes. Compare signed errors and innovation patterns as well as coverage and uncertainty width. If ground truth is available, keep its acquisition independent of the estimator being assessed.

Choose a motion-model change or noise-statistics adjustment for a stated reason, then repeat the evaluation. One covered fresh case is useful feedback, but it does not prove calibration across conditions.

A repair should earn its improvement on trajectories that were not used to choose its parameters.