The Detector Found the Object. Where Is It for the Robot?

Geometry Connects Separate Sensors

BOTIE

An arm joint describes motion relative to another joint. A camera describes an object relative to the camera. Giving both descriptions numbers does not make their origins and directions agree. Mechanical systems needed a consistent way to connect these local descriptions before a learned detector entered the picture.

Denavit and Hartenberg's 1955 paper described mechanisms using matrix relationships. Writing each relationship explicitly made it possible to combine a chain of links through matrix algebra. The useful idea here is a change of coordinates: the object stays where it is while we express its position using another origin and set of axes.

We will use a supplied rotation and translation to turn a detector's position into the robot's coordinates. This known rigid transform is simpler than estimating calibration or the robot's location. It lets us inspect the geometric step between recognising an object and choosing where the robot should move.

Sources for this history

Left of the Camera Is Not Always Left of Botie

A perception model reports an object two metres ahead of the camera. Botie’s camera is mounted sideways and away from the robot’s origin. Sending those coordinates directly to a motion planner would point at the wrong place.

A coordinate is a measurement relative to an origin and a set of directions. We need to change that reference before using the prediction.

Known exact planar calibration; no pixel-to-depth inference or physical actuation occurs.

Step 1 of 1

A camera-relative point needs two changes

Rotation changes the directions; translation changes the origin.

After rotation the point is [0,2]; translation makes it [1,5].
Quantity	Teaching value
Camera point	[2,0]
Rotation	90° counterclockwise
Camera origin in robot frame	[1,3]

Camera point
Teaching value
[2,0]
Rotation
Teaching value
90° counterclockwise
Camera origin in robot frame
Teaching value
[1,3]

After rotation the point is [0,2]; translation makes it [1,5].

Name the direction of the transform

Rotation and translation have different roles and a declared direction.

Connect perception to the robot frame

The camera’s directions differ from the robot’s directions.

BOTIE

The camera’s directions differ from the robot’s directions.

Choose an example

Camera-to-robot rotation

1

2

1

2

Camera point in metres

1

2

Camera origin in robot frame

1

2

Point in robot coordinates

03
13

Follow the operations

1. Camera-to-robot rotation
1 0
0 1
2. Camera point in metres
1. 02
2. 10
3. Matrix times vector
1. 02
2. 10
4. Camera origin in robot frame
1. 01
2. 13
5. Add
1. 03
2. 13
6. Result
1. 03
2. 13

Known exact planar calibration; no pixel-to-depth inference or physical actuation occurs.

1. Name the direction of the transform

Write down that this transform maps camera coordinates into robot coordinates. Its rotation expresses camera axes in the robot frame. Its translation is the camera origin expressed in the robot frame.

An inverse transform answers the opposite question. Reusing the same matrix in both directions generally gives the wrong point.

Pixel coordinates are not metre coordinates. A detector that returns pixels needs camera geometry and depth or another geometric assumption before this step.

2. Rotate, then add the origin offset

Use a matrix-vector multiplication for the rotation. Add the translation afterwards because it is already expressed in the destination frame.

R = [[0,-1], [1,0]]
p_camera = [2,0]
t_robot_camera = [1,3]
p_robot = R @ p_camera + t_robot_camera = [1,5]

Applying the translation before rotation rotates the offset too, which describes a different transform.

Rotate a direction before moving the origin

A two-dimensional vector is a one-dimensional NumPy array with two entries.

Python

import numpy as np
rotation = np.array([[0,-1],[1,0]])
point = np.array([2,0])
print((rotation @ point).tolist())

Output

[0, 2]

The translation has not been applied yet. This result is only the rotated point relative to the camera origin.

Check yourself

Which frame owns this point?

Question 1 of 2

Where is the point in the robot frame?

Use the information below.

R=[[0,-1],[1,0]], p=[2,0], t=[1,3].

3. The frame has a time as well

A camera on a moving arm changes pose. Use the transform appropriate to the observation timestamp. Combining an old image with the latest pose can create an error even when both are individually correct.

Uncertainty also changes coordinates. For a fixed known rotation, covariance transforms with the rotation on both sides; uncertain calibration adds another source of error.

This workshop treats calibration as exact. The next lesson handles a separate timing approximation rather than pretending coordinates alone solve motion.

4. Test geometry before connecting an actuator

Identity rotation and zero translation should preserve a point. A pure translation should shift it by exactly that amount. Known right-angle rotations expose swapped axes and sign errors.

Use simulation and independent physical limits before a learned position can command motion. A correct transform cannot repair a detector that found the wrong object.

The exercise transforms numerical points only and controls no hardware.