Separating a Useful Prediction From a Causal Claim

Plants receiving extra water may grow poorly because they were already struggling. Compare the question a prediction answers with the question an intervention asks.

13 sections · 4 questions · 2 coding challenges

1 / 13

The arrow keys move between sections too.

Making the Intervention Question Explicit

BOTIE

Causal modelling distinguishes observing a choice from intervening to set it. Pearl’s graphical framework makes assumptions about common causes and pathways explicit, allowing us to ask when observational comparisons identify an intervention effect.

Our plant example is invented. Its purpose is to expose why a common cause of treatment and outcome can distort a comparison, not to establish an agricultural recommendation.

Sources for this history

Pearl: Causality, second edition

Did Extra Water Hurt the Plants?

Botie notices that plants given extra water grew less. It would be easy to conclude that watering caused the poor growth. But the gardener may have watered the struggling plants more often.

The treatment groups began differently. We need to understand that selection before asking what would happen if we changed the watering rule.

The garden outcomes are invented to demonstrate confounding by starting condition.

Step 1 of 3

Notice unequal starting conditions

Extra water is given more often to dry plants.

Extra-water group: mostly dry beds
Usual-water group: mostly moist beds

Step 2 of 3

Compare within starting conditions

The hypothetical treatment mean exceeds control in each stratum.

An invented example for this lesson.
Starting soil	Extra water	Usual water
Dry: treatment, control	4	3
Moist: treatment, control	9	8

Dry: treatment, control
Extra water
4
Usual water
3
Moist: treatment, control
Extra water
9
Usual water
8

Step 3 of 3

Inspect the mixture

Different group proportions can reverse the pooled comparison.

The pooled groups give very different weights to the starting soil conditions. Their overall means can reverse the within-soil comparisons.
Group	Dry share	Moist share	Pooled mean
Extra water	0.9	0.1	(0.9 × 4) + (0.1 × 9) = 4.5
Usual water	0.1	0.9	(0.1 × 3) + (0.9 × 8) = 7.5

Extra water
Dry share
0.9
Moist share
0.1
Pooled mean
(0.9 × 4) + (0.1 × 9) = 4.5
Usual water
Dry share
0.1
Moist share
0.9
Pooled mean
(0.1 × 3) + (0.9 × 8) = 7.5

The pooled groups give very different weights to the starting soil conditions. Their overall means can reverse the within-soil comparisons.

Group composition can change the conclusion of a pooled comparison

The calculation makes each group’s composition visible.

Are we comparing schedules or different kinds of beds?

Inspect the outcome within each starting condition, then change the proportions of dry and moist beds in the two groups. The pooled comparison can move without any within-condition result changing.

BOTIE

Botie has means within dry and moist beds and each group’s own proportions.

Choose an example

Treatment stratum means

Control stratum means

Treatment proportions

Control proportions

Pooled treatment mean minus pooled control mean

-3

Follow the operations

1. Treatment stratum means
1. 04
2. 19
2. Treatment proportions
1. 00.9
2. 10.1
3. Check a probability distribution
1. 00.9
2. 10.1
4. Dot product
```
4.5
```
5. Control stratum means
1. 03
2. 18
6. Control proportions
1. 00.1
2. 10.9
7. Check a probability distribution
1. 00.1
2. 10.9
8. Dot product
```
7.5
```
9. Subtract
```
-3
```
10. Result
```
-3
```

This is an observed difference. A causal interpretation requires additional design and assumptions.

1. State the Two Different Questions

A prediction asks what outcome is likely among records with certain measurements. A causal question asks how the outcome would change under a specified intervention. A variable can help predict an outcome without being a useful lever to change it.

Define treatment, outcome, population, and observation window. Extra water needs a quantity and schedule before it becomes a well-defined intervention.

The example studies a comparison procedure, not a claim that more water always helps.

2. Identify a Common Cause

Starting soil condition could influence both the gardener's watering choice and the plant's growth. That makes it a possible confounder: the groups may differ for a reason that existed before treatment.

A correlation alone does not establish that role. We need knowledge of how the measurements and decisions arose, including their timing. A feature that predicts growth well is not automatically something we should adjust for.

Measuring a variable after treatment can give it a very different role.

3. Inspect Group Composition

Calculate pooled means by weighting stratum means with each group’s own stratum proportions. Then compare the same stratum across groups. A reversal shows that the pooled association depends strongly on composition.

Treatment mean = 0.9 × 4 + 0.1 × 9 = 4.5
Control mean = 0.1 × 3 + 0.9 × 8 = 7.5
Pooled difference = -3
Within-stratum differences = [1,1]

A reversal does not itself prove the adjusted comparison is causal. The relevant assumptions still need support.

A benefit within each group can reverse after pooling

The treatment outcome is higher within both starting conditions. However, the treatment group contains mostly the harder condition and the control group mostly the easier one.

Python

import numpy as np
treatment = np.array([2., 8.])
control = np.array([0., 6.])
treatment_mix = np.array([0.8, 0.2])
control_mix = np.array([0.2, 0.8])
print((treatment - control).tolist())
print(f'{treatment @ treatment_mix:.1f}')
print(f'{control @ control_mix:.1f}')

Output

[2.0, 2.0]
3.2
4.8

The pooled treatment average is lower despite both within-condition differences being positive. Group composition changes the comparison we are making.

4. Choose Evidence That Matches the Claim

Random assignment can remove systematic treatment selection under a valid experiment. Observational adjustment can be justified when measured variables suffice to block confounding and other identification conditions hold.

More predictive features do not automatically produce a better causal estimate. Conditioning on an effect of treatment or on a collider can create a different bias.

The next lessons establish experimental comparisons and then a carefully stated observational adjustment.

5. Compare Like Starting Conditions before Combining Them

If the new watering schedule is used more often on dry beds, its overall growth average mixes treatment differences with differences in starting soil. Compare schedules within the same pre-treatment soil group first.

Then combine those comparisons using one shared target population. Giving each treatment its own soil mixture recreates the original imbalance rather than removing it.

Separate group effects from group composition

Both soil groups gain two units under the new schedule, but treated beds are mostly in the harder dry group.

Python

import numpy as np
old = np.array([2.,8.])
new = np.array([4.,10.])
old_mix = np.array([.2,.8])
new_mix = np.array([.8,.2])
target_mix = np.array([.5,.5])
print(round(float(new_mix@new-old_mix@old), 2))
print(round(float(target_mix@(new-old)), 2))

Output

-1.6
2.0

The unadjusted difference reverses the within-group comparison in this invented table. The adjusted arithmetic has a causal interpretation only under the required assumptions.

6. Do Not Control for a Variable Merely because It Predicts the Outcome

A confounder is a common cause of treatment and outcome. A mediator lies on a pathway through which treatment affects outcome. Adjusting for a mediator changes the effect being estimated and can introduce other problems.

A collider is affected by two variables. Conditioning on it can create an association that was absent before. Draw the assumed directions from the problem before selecting adjustment columns. Predictive usefulness alone does not identify an appropriate adjustment set.

7. An Adjustment Cannot Repair an Unmeasured Common Cause

Soil condition may be recorded too crudely, or the gardener may use unrecorded plant health when assigning extra water. Comparing recorded soil groups does not guarantee comparable treatment groups within them.

State what would need to be measured and what remains uncertain. Use domain knowledge, study design and sensitivity analysis to examine those assumptions. A clean numerical difference is not proof that every relevant cause was controlled.