Valuing Rewards That Arrive Later

An action can cost water now and help a plant later. A return combines a sequence of rewards so Botie can compare more than the immediate outcome.

13 sections · 4 questions · 3 coding challenges

1 / 13

The arrow keys move between sections too.

Connecting Immediate Feedback to Later Outcomes

BOTIE

Sequential prediction needed to account for outcomes that arrive after several intermediate steps. Sutton's 1988 work developed temporal-difference learning, where later predictions help improve earlier predictions before a final outcome is available.

Before using that update, we need to define the quantity predicted. A discounted return combines future rewards with a stated time convention. Our short watering episode makes the accumulation explicit.

Sources for this history

Would Botie Reject a Useful Action Because It Costs Something Now?

Watering costs resources immediately, while healthier growth appears later. If Botie judges only the immediate reward, the useful action can look worse than doing nothing.

We need a way to judge an action using rewards that arrive later too. A return combines rewards over time. Discounting controls how much a delayed reward contributes compared with an immediate one. Before calculating it, we should ask whether that preference matches the job we want Botie to do.

The short reward sequence is an illustrative completed episode.

Step 1 of 3

Separate the steps

The action costs one unit now and leads to a later reward in this invented completed episode.

An invented example for this lesson.
Reward time	Rewards in order
Now	-1
One step later	0
Two steps later	4

Now
Rewards in order
-1
One step later
Rewards in order
0
Two steps later
Rewards in order
4

Step 2 of 3

Choose the discount

With gamma one half, later contributions receive successively smaller weights.

An invented example for this lesson.
Reward time	Weights
Now	1
One step later	0.5
Two steps later	0.25

Now
Weights
1
One step later
Weights
0.5
Two steps later
Weights
0.25

Step 3 of 3

Combine the contributions

The discounted return is zero under this particular weighting, even though the undiscounted total is positive.

Now: −1 × 1 = −1
One step later: 0 × 0.5 = 0
Two steps later: 4 × 0.25 = 1
Return = −1 + 0 + 1 = 0

A later reward contributes less under this chosen discount. Add contributions only after weighting them.

A return combines rewards according to when they arrive

Each extra delay introduces another factor of gamma. The dot product combines the weighted contributions.

How much does waiting change the value of a reward?

Keep the reward sequence fixed and change the discount. Follow the contribution of a later reward and compare it with one available immediately.

BOTIE

Botie has a completed sequence of rewards and a discount per step.

Choose an example

Rewards after successive actions

Discount per step

Discounted return from the start

Follow the operations

1. Rewards after successive actions
1. 0-1
2. 10
3. 24
2. Discount per step
```
0.5
```
3. Build discount weights
1. 01
2. 10.5
3. 20.25
4. Dot product
```
0
```
5. Result
```
0
```

Rewards and episode termination are supplied. A return from one realised episode is not a policy's expected value.

1. Define the Reward Sequence Before Adding It

An episode is a sequence that ends at a terminal condition. Its return starts with the reward received after the current action. In our arrays, the first entry is that immediate reward, not a reward from before the decision.

The reward design and episode boundary affect what is optimised. Cutting off a simulation for convenience is not necessarily the same as reaching a genuine terminal state.

Rewards after the current action: r[0], r[1], r[2], ...

State clearly whether the final reward has already been included.

2. Apply One Discount for Each Additional Delay

Gamma controls the weight retained per time step. Values between zero and one emphasise earlier rewards; gamma zero ignores later rewards. Gamma one gives an undiscounted sum for a finite episode.

Discounting can encode a preference over delay and helps define finite returns in continuing problems under suitable conditions. It is not universally the physical probability that a plant survives each day.

G = r[0] + gamma × r[1] + gamma^2 × r[2]
For [-1,0,4] and gamma=0.5:
G = -1 + 0.5 × 0 + 0.25 × 4 = 0

Gamma controls how much future rewards contribute to the return. Changing it can change which behaviour the agent prefers.

3. Work Backward Through a Completed Episode

The return at a position equals its immediate reward plus gamma times the return beginning at the next position. At the end of a completed finite episode, there is no remaining reward.

A backward loop calculates every position's return without recomputing every suffix separately. These are realised returns from one episode; a value function predicts their expectation under a policy.

G[t] = r[t] + gamma × G[t+1]
Beyond the terminal step: G = 0

This recurrence is the connection to the Bellman calculations in the next lesson.

See each reward's contribution before adding

These rewards arrive now, one step later and two steps later. Give each delay its own power of the discount before forming the return from the start.

Python

import numpy as np
rewards = np.array([2., 0., 8.])
gamma = 0.5
weights = gamma ** np.arange(len(rewards))
contributions = weights * rewards
print(weights.tolist())
print(contributions.tolist())
print(float(contributions.sum()))

Output

[1.0, 0.5, 0.25]
[2.0, 0.0, 2.0]
4.0

The distant reward contributes less because it is delayed twice. The challenge asks for the return starting at every position, which is easier to calculate backwards.

4. Separate Observed Returns From Estimated Values

Two episodes under the same policy can produce different returns because transitions and rewards may be random. A single successful episode does not establish that a policy has high expected value.

Compare policies across repeated trials or a known transition model, keeping the reward definition and discount fixed. Botie's construction combines a supplied episode's rewards with its discount weights.

The workshop reports a realised discounted return. It neither learns a policy nor estimates an expectation from one trace.

5. Calculate a Return for Every Decision in the Episode

The first action needs all later rewards, but later actions need only their remaining suffix. Starting from the episode's end lets us reuse the return already calculated for the next position.

The result contains one return per action time. It is not one reward copied onto every decision, and it still reflects the actions actually taken later in that episode.

Propagate the harvest reward backward

The episode pays two small costs before a final benefit. Discount is one half.

Python

import numpy as np
rewards = np.array([-1., -1., 8.])
returns = np.zeros_like(rewards)
future = 0.
for t in range(len(rewards)-1, -1, -1):
    future = rewards[t] + .5*future
    returns[t] = future
print(returns.tolist())

Output

[0.5, 3.0, 8.0]

The first decision includes both later costs and benefits at their respective delays.

6. A Time Limit Is Not Always the End of the Task

A true terminal state has no future reward under the task definition. A training rollout that stops because a time limit was reached may still represent an ongoing process. Setting the continuation to zero then changes the target.

For a continuing task, a truncated rollout can use an estimated value at its final state. State clearly whether the time limit is part of the environment's objective or merely a sampling limit. This distinction carries directly into temporal-difference targets.

7. Observed Returns Can Vary under the Same Policy

Different transitions and action samples can produce different returns from the same state. Averaging repeated returns estimates a policy value under those conditions, but long episodes can have substantial variability.

Bootstrapped methods use a current value estimate for part of the future, trading some dependence on that estimate for shorter sampled targets. Neither the discounted sum nor one successful episode establishes that a policy is optimal.