Reweighting Records Without Inventing Missing Evidence

Inverse-probability weighting changes which records count most. Understand the treatment probabilities and overlap assumptions before interpreting the resulting contrast.

14 sections · 4 questions · 2 coding challenges

1 / 14

The arrow keys move between sections too.

Making Observed Groups Represent a Shared Population

BOTIE

The potential-outcomes framework defines an average effect as a comparison of outcomes under two interventions. Hernán and Robins develop identification and estimation methods, including weighting, while making exchangeability, positivity, and consistency assumptions explicit.

Our supplied propensity scores are part of a teaching model. Estimating them from observations adds uncertainty and model dependence; weighting does not automatically remove unmeasured confounding.

Sources for this history

How Can Rarely Treated Beds Represent Similar Untreated Ones?

Some types of bed rarely received the new schedule. If one of those beds did receive it, its outcome carries information about a group underrepresented among treated records.

Inverse-probability weighting gives more weight to such records. That idea is useful only when treatment probabilities are credible and both schedules remain possible for the population being studied.

The supplied propensities must lie strictly between zero and one.

Step 1 of 3

Record assignment probabilities

A propensity is the probability of treatment given pretreatment covariates.

An invented example for this lesson.
Observed bed	Treatment indicators	Propensities
Bed 1	1	0.25
Bed 2	0	0.75

Bed 1
Treatment indicators
1
Propensities
0.25
Bed 2
Treatment indicators
0
Propensities
0.75

Step 2 of 3

Weight the observed branch

Treatment uses its own probability; control uses the complement.

An invented example for this lesson.
Observed bed	Observed-branch weights
Bed 1	4
Bed 2	4

Bed 1
Observed-branch weights
4
Bed 2
Observed-branch weights
4

Step 3 of 3

Inspect extreme values

Very small observed-branch probabilities create large weights.

Probability of the observed assignment: 0.01
Weight = 1 / 0.01 = 100

A rare observed assignment receives a large inverse-probability weight. That observation can then dominate the estimate.

Weighting cannot replace evidence where overlap is missing

Weighting changes how observed outcomes represent the target population under the identification assumptions.

Which records carry most of the weighted comparison?

A treatment received despite a very small propensity can receive a large weight. Change a valid propensity and inspect its influence before trusting the final contrast.

BOTIE

Botie has outcomes, binary treatment assignments, and valid supplied propensities.

Choose an example

Binary treatment assignments

Observed outcomes

Propensities strictly between zero and one

Inverse-probability weighted treatment contrast

2.5

Follow the operations

1. Binary treatment assignments
1. 01
2. 10
3. 21
4. 30
2. Observed outcomes
1. 06
2. 14
3. 28
4. 35
3. Multiply
1. 06
2. 10
3. 28
4. 30
4. Propensities strictly between zero and one
1. 00.5
2. 10.5
3. 20.5
4. 30.5
5. Check treatment overlap
1. 00.5
2. 10.5
3. 20.5
4. 30.5
6. Divide
1. 012
2. 10
3. 216
4. 30
7. Set a constant
```
1
```
8. Subtract
1. 00
2. 11
3. 20
4. 31
9. Multiply
1. 00
2. 14
3. 20
4. 35
10. Subtract
1. 00.5
2. 10.5
3. 20.5
4. 30.5
11. Divide
1. 00
2. 18
3. 20
4. 310
12. Subtract
1. 012
2. 1-8
3. 216
4. 3-10
13. Mean
```
2.5
```
14. Result
```
2.5
```

This uses the unnormalised all-unit mean. Positivity and no unmeasured confounding cannot be established by the arithmetic.

1. Name the Missing Outcomes

For each bed, imagine its outcome under the new schedule and its outcome under the control schedule. We observe only the one corresponding to what it received. The average treatment effect concerns the average difference between those potential outcomes, even though we usually cannot measure an individual bed's difference directly.

The propensity e(X) describes the probability of receiving treatment given pretreatment measurements. It says how the schedule was assigned. It is not a predicted growth outcome or a confidence score for that prediction.

A propensity model predicts assignment, while an outcome model predicts the response.

2. Calculate Weights for the Observed Branch

A treated unit receives inverse propensity. A control unit receives inverse one minus propensity. Under the identification assumptions, these weights can make the groups represent a shared covariate distribution.

The unnormalised Horvitz-Thompson-style difference averages weighted outcome contributions over all n units. A normalised group-weight estimator is a related but different finite-sample calculation. State which one is used.

Contribution_i = T_i × Y_i / e_i - (1-T_i) × Y_i / (1-e_i)
ATE estimate = mean(Contribution_i)

The exercises use the unnormalised all-unit mean, not separately normalised weighted group means.

3. Require Overlap Before Trusting the Estimate

If a relevant covariate group always receives one treatment, the other potential outcome has no observed support there. Propensities near zero or one can also produce highly variable estimates from a few large weights.

Inspect weight distributions and covariate balance. Trimming or clipping weights changes the estimation procedure and can change the target or introduce bias. It is not a silent numerical repair.

Exact zero or one propensities are rejected in the workshop.

Inspect the weighted contributions before averaging

One treated record had a treatment propensity of one quarter. One control record had a treatment propensity of one half. Use the probability of the treatment each actually received.

Python

import numpy as np
treatment = np.array([1., 0.])
outcome = np.array([4., 2.])
propensity = np.array([0.25, 0.5])
contributions = treatment * outcome / propensity - (1 - treatment) * outcome / (1 - propensity)
print(contributions.tolist())
print(float(contributions.mean()))

Output

[16.0, -4.0]
6.0

The first record has much greater influence. This is why a final weighted contrast needs an overlap check as well as correct arithmetic.

4. Keep Causal and Statistical Uncertainty Separate

A precise numerical estimate can still be biased by an omitted common cause, ill-defined treatment, interference, or a misspecified propensity model. Sampling uncertainty does not account automatically for these identification failures.

Use domain knowledge, design checks, overlap diagnostics, and sensitivity analysis. If the assumptions cannot be defended, report an association or restricted comparison rather than presenting a causal answer as established.

Botie’s workshop exposes weighting arithmetic without claiming that supplied propensities prove identification.

5. A Large Weight Is a Warning about Thin Evidence

If a kind of bed rarely receives the new schedule, a treated bed of that kind represents many similar beds under inverse-probability weighting. Its outcome can therefore dominate the estimate.

Inspect propensity ranges and weight concentration separately for treated and untreated records. A fitted probability near zero or one may reflect genuine lack of overlap, model misspecification or a small sample. None is fixed merely by printing more decimal places.

6. Measure How Concentrated the Weights Are

The effective sample size below describes how uneven weights reduce the information represented by a weighted average. It is a diagnostic, not a complete uncertainty calculation.

Compare it with the raw number of records. A large dataset whose result depends on a few enormous weights may still provide fragile evidence for the target population.

Four records with one dominant weight

These are supplied nonnegative weights for one comparison group.

Python

import numpy as np
weight = np.array([1.,1.,1.,9.])
effective = weight.sum()**2/(weight@weight)
print(len(weight))
print(round(float(effective), 3))

Output

4
1.714

Four records have the weight concentration of far fewer equally weighted records under this diagnostic.

7. Normalising Weights Changes the Finite-Sample Estimator

One estimator averages inverse-weighted outcomes over the full sample size. Another divides each group's weighted outcome sum by that group's weight sum, then takes the difference. The latter is often called a normalised or Hájek form.

These estimators can differ in finite samples even under the same propensity model. State which formula is used. Clipping large weights also changes the procedure and can trade variance for bias; it is not a neutral numerical repair.

8. Overlap Is Necessary but Not Sufficient

Even well-behaved weights do not establish that all treatment-outcome common causes were measured. Identification also depends on consistency and the stated exchangeability assumptions, with interference considered where relevant.

Report the target population, assignment model, balance checks and remaining causal assumptions. A predictive model of treatment assignment can help estimate propensities, but its classification accuracy is not the causal objective.