The Policy Works in One Simulator Setting

Variation Can Help a Model Cross the Reality Gap

BOTIE

A visual model can learn the lighting or textures of a simulator so well that it struggles with real images. Making the simulator look realistic in one particular setting does not tell us which details the model is relying on.

Tobin and colleagues' 2017 domain-randomization work varied rendering conditions while training a model for robotic object localisation. By seeing many appearances, the model had less reason to depend on one simulated appearance. The researchers studied whether that training could transfer to real images. This was an experimental response to the gap between simulated and real observations, not a guarantee that arbitrary variation would solve it.

Botie's policy-cost table applies the broader comparison idea to a smaller problem. We will change the supplied conditions and see whether a policy still behaves acceptably. The original paper studied visual transfer; our table does not reproduce that experiment or establish safety in an unseen physical setting.

Sources for this history

Tobin and colleagues: Domain Randomization, 2017

Botie Learned the Clean Floor Too Well

A policy succeeds on the simulator’s default floor. Change friction and it starts missing the target. Training on one convenient setting let the policy depend on a condition we never intended to guarantee.

We can vary plausible conditions during training and evaluate on separate conditions. We also need to keep each result visible. An average can hide the setting where the policy becomes unusable.

Invented condition costs; the maximum is worst tested, not a guarantee over all physical conditions.

Step 1 of 1

Keep the condition beside the score

Lower cost is better; the acceptance ceiling is declared separately.

The worst tested condition matters even when the mean seems acceptable.
Condition	Candidate cost
Dry floor	2
Light load	3
Low friction	12

Dry floor
Candidate cost
2
Light load
Candidate cost
3
Low friction
Candidate cost
12

The worst tested condition matters even when the mean seems acceptable.

Define which variation the deployment can encounter

The summary keeps a difficult condition from disappearing into the average.

Find the condition an average conceals

Botie’s learned policy fails when a simulated physical condition changes.

BOTIE

Botie’s learned policy fails when a simulated physical condition changes.

Choose an example

Policy costs across declared conditions

1

2

3

Mean cost and worst tested condition cost

03
14

Follow the operations

1. Policy costs across declared conditions
1. 02
2. 13
3. 24
2. Mean
```
3
```
3. Largest value
```
4
```
4. Collect two results
1. 03
2. 14
5. Result
1. 03
2. 14

Invented condition costs; the maximum is worst tested, not a guarantee over all physical conditions.

1. Define which variation the deployment can encounter

Identify uncertain parameters such as friction, mass, lighting or sensor delay. Use measurements and domain knowledge to choose plausible ranges. Independent randomisation can create impossible combinations if physical parameters are related.

Separate the training distribution from evaluation conditions. Include meaningful corners and combinations rather than only draws close to defaults.

A simulator can omit an entire failure mechanism. Sampling more values of existing parameters cannot reveal a mechanism that is absent from the model.

2. Report both average and worst tested cost

The mean summarises these tested conditions under equal weighting. The maximum preserves the worst observed case. Neither is automatically the expected deployment cost without a justified weighting distribution.

costs = [2,3,12]
mean = 17 / 3
worst tested = 12

Worst tested is not worst possible. Untested conditions may be worse, especially outside the simulator’s assumptions.

Retain condition identities when a requirement fails

Keep names alongside costs so a failure leads to a reproducible scenario.

Python

names = ["dry", "light", "slippery"]
costs = [2,3,12]
print([name for name,cost in zip(names,costs) if cost > 8])

Output

['slippery']

The next repair can target a concrete condition while a later assessment uses fresh evidence.

Check yourself

What did the tested conditions actually establish?

Question 1 of 2

What is the worst tested cost?

Use the information below.

Condition costs [2,3,12].

3. Keep a requirement separate from a ranking

Choose a cost ceiling or other constraint before reviewing the candidate. A lower mean than baseline can still fail that requirement in one condition. Retain the failed condition for diagnosis, then assess the repair on fresh conditions.

For stochastic policies, repeat scenarios sufficiently to distinguish condition effects from random variation. The paired-trial lesson explains the unit of comparison.

A deterministic table is enough to inspect the decision logic but not to estimate a real failure probability.

4. Move toward the real system with new evidence

Validate observations and transitions against held-out measurements. Use controlled trials and physical safeguards appropriate to the system. A simulated passing score is evidence about the simulator, with transfer still to establish.

If a real trial exposes a missing effect, revise the simulator and preserve the earlier benchmark as regression evidence. Do not describe the revised simulator’s own training data as independent validation.

The workshop never sends actions to hardware. It helps decide which simulated failures need investigation before any transfer claim.