Give Two Policies the Same Simulated Shift

Simulation Is an Experiment with a Random Input

BOTIE

Suppose we test two dispatch policies in simulation. One receives a quiet shift while the other receives a rush of difficult orders. Their different costs now include the different conditions, making the policy change harder to judge. Simulation researchers used shared random inputs to compare alternatives under related conditions.

Glasserman and Yao examined this established technique in their 1992 paper on common random numbers. Their analysis asked when sharing randomness actually improves a comparison. The benefit comes from how the paired outcomes move together; merely reusing a seed is not a guarantee of a better estimate.

Our exercise makes the pairing explicit by giving both policies the same simulated shift. We compare their costs within that shift before combining evidence across shifts. A reproducible generator helps create the scenarios, but matching the conditions that matter is the experimental design decision.

Sources for this history

Did the Policy Improve, or Did It Get an Easier Shift?

Botie tests two inspection policies in a simulator. One run happens to contain fewer difficult faults. Its lower cost might come from the policy or from the luck of the generated shift.

We can give both policies the same scenario and compare their costs within it. Then repeat over independently sampled scenarios. The pairing preserves which difficulty they shared.

Costs are invented. Statistical uncertainty needs independent replications and physical transfer needs separate validation.

Step 1 of 1

Keep the costs paired by shift

Positive gain means the candidate costs less.

The unusual shift regresses even though the candidate improves the mean.
Shift	Baseline cost	Candidate cost
Easy	10	9
Busy	30	25
Unusual	20	22

Easy
Baseline cost
10
Candidate cost
9
Busy
Baseline cost
30
Candidate cost
25
Unusual
Baseline cost
20
Candidate cost
22

The unusual shift regresses even though the candidate improves the mean.

Sample a scenario before choosing the policy

The subtraction retains shared scenario difficulty before averaging.

Compare policies on the same simulated shifts

Botie must separate policy changes from differences in scenario difficulty.

BOTIE

Botie must separate policy changes from differences in scenario difficulty.

Choose an example

Baseline cost per scenario

1

2

3

Candidate cost for the same scenarios

1

2

3

Mean cost reduction on shared scenarios

Follow the operations

1. Baseline cost per scenario
1. 010
2. 120
3. 230
2. Candidate cost for the same scenarios
1. 08
2. 118
3. 228
3. Subtract
1. 02
2. 12
3. 22
4. Mean
```
2
```
5. Result
```
2
```

Costs are invented. Statistical uncertainty needs independent replications and physical transfer needs separate validation.

1. Sample a scenario before choosing the policy

A scenario can contain fault arrivals, sensor noise and available staff. Keep the conditions that neither policy controls fixed for the comparison. These are the exogenous conditions: for example, the same incoming faults rather than a fresh, easier set for the second policy. If policy actions change future events, define a simulator coupling that remains meaningful rather than forcing incompatible trajectories to match.

Use independent scenarios for uncertainty calculations. Repeated steps inside one simulated shift are usually dependent.

A seed identifies random state under an implementation; it is not a complete record of simulator version or parameters.

2. Subtract within each scenario

Define gain as baseline cost minus candidate cost. Positive gain means a cost reduction. Average the per-scenario gains only after preserving their pairing.

baseline = [10,30,20]
candidate = [9,25,22]
gains = [1,5,-2]
mean gain = 4 / 3

Keep regressions visible. An improved average can still violate a requirement for a particular operating condition.

Inspect the paired changes before reducing them

Arrays retain the shared scenario order.

Python

import numpy as np
baseline = np.array([10,30,20])
candidate = np.array([9,25,22])
print((baseline-candidate).tolist())

Output

[1, 5, -2]

The negative third gain is a regression on that shared scenario and should remain visible.

Check yourself

Which comparison preserves the shared difficulty?

Question 1 of 2

Which gains preserve pairing?

Use the information below.

Baseline [10,30,20], candidate [9,25,22].

3. Report how uncertain the estimated gain is

For independent paired scenarios, estimate standard error from the sample standard deviation of the gains divided by the square root of the number of scenarios. At least two scenarios are needed for this sample variance.

A standard error is not a guarantee that the simulator represents reality. It quantifies sampling variation under the stated simulation and independence assumptions.

For correlated replications, use an appropriate replication or blocking design rather than pretending every step is independent.

4. Separate development seeds from assessment scenarios

Reusing development seeds helps debug a policy. Repeatedly tuning to those seeds can overfit their particular events. Fix the candidate before evaluating fresh scenario draws.

Record the simulator, parameters and random generator configuration with results. A different simulator revision can change the meaning of a repeated seed.

This lesson uses supplied finite costs so the comparison is inspectable; it does not claim a real-world policy improvement.