A Finite Vector Broke Our Normalisation

Botie compares a query with documents represented by lists of numerical coordinates, called vectors. For this comparison we care about their direction while setting aside their overall size. The small example works. Then the encoder, the model producing those coordinates, returns much larger components, and the same calculation produces zeros.

The direction did not disappear from the input. We lost it while trying to calculate the vector’s length. Let’s follow those operations before deciding that the model gave us a useless result.

Botie’s retrieval pipeline turns a valid large embedding into zeros while trying to normalise it.

Picture two arrows pointing in the same direction, with one longer than the other. The coordinate lists [3,4] and [6,8] describe that relationship. When our retrieval comparison uses direction, we want both lists to produce the same length-one vector. Dividing every component by one common positive length preserves their proportions.

The coordinate positions must still mean the same things. Position zero in the query must match position zero in a document from that encoder. Removing length cannot make unrelated encoder coordinate systems comparable.

length(x) = sqrt(sum of the squared components)
unit(x) = x / length(x)

For an all-zero vector there is no direction to preserve. We return zeros as an explicit special outcome, so the caller can decide how to handle that document.

A floating-point number can be finite while its square is too large to store. At the opposite end, a small nonzero square can round to zero. Neither outcome establishes that the input itself was invalid.

Inspect intermediate values and the array dtype. Switching to a wider type may postpone a range problem, but an unsuitable evaluation can still fail. Replacing an infinity with a large constant would also change the result without recovering the lost calculation.

large components: [3e200, 4e200]
direct squares: [9e400, 16e400]
these squares exceed binary64 range

The mathematical squares exist. The machine format cannot represent them as finite binary64 values.

Use the largest absolute component as a common scale. After dividing by that scale, every component has magnitude at most one. Now compute the length of this scaled vector and divide by it.

We do not need to multiply back by the original scale when the goal is a unit direction. Avoiding that unnecessary multiplication also helps when the original mathematical length itself would exceed the representable range.

scale = max(abs(x))
z = x / scale
unit(x) = z / sqrt(sum(z * z))

x = [3e200, 4e200]
z = [0.75, 1]
length(z) = 1.25
unit(x) = [0.6, 0.8]

Each operation has a reason. The first division protects the products; the second removes the remaining length.

A document can produce a nonempty vector whose entries are all zero. Its magnitude is zero, so it has no direction to preserve. Our procedure returns a vector of zeros in that case. It rejects an empty vector, which has no components at all. An application must still decide whether to reject a document represented by zeros, skip it or use a different representation.

In the workshop, a zero magnitude is replaced with one only when choosing a divisor. That leaves the zero values at zero. We are not asserting that their direction is now known.

Using a very small arbitrary constant as every divisor can distort small, meaningful vectors. Handle the actual zero case separately.

Follow one large vector through the program below. The scaled components form a new NumPy array. Taking that array’s dot product with itself gives one number, and the final division applies the same divisor to every component.

Pause at the printed scale and scaled coordinates. This is where we avoided forming the enormous squares of the original values.

The returned coordinates can now be used for the intended direction comparison. Whether the retrieved document is useful still needs its own evaluation.

import numpy as np
x = np.array([3e200,4e200])
z = x / np.max(np.abs(x))
print(z.tolist())
print(np.round(z / np.sqrt(z @ z),3))

[0.75, 1.0]
[0.6 0.8]

A table of embeddings usually has one example per row and one coordinate per column. Find a separate scale for every row. Keeping the reduced axis as a length-one dimension lets NumPy broadcast each scale back across its own row.

One scale for the entire table is mathematically possible in exact arithmetic, but it can lose the small rows when another row is vastly larger. Independent row scaling matches the calculation we actually want.

batch shape: (number_of_documents, encoder_dimensions)
row scales shape with keepdims=True: (number_of_documents, 1)

Axes are part of the data contract. Normalising each column answers a different question.

Test large and tiny positive rescalings of the same nonzero vector. The unit direction should stay close. Test sign changes, zero rows and malformed inputs as separate behaviours.

Comparing the norm alone is insufficient: many wrong directions have length one. Keep an expected direction for a simple vector whose geometry we can inspect.

This fixes one source of numerical failure. A drifting encoder, an incorrect similarity metric and a poor retrieval corpus still require their own evidence.