Compare immediate rewards and future states

Give the future more weight

Botie must compare an immediate reward with the value of where each action leads. The discount can change that choice.

BOTIE

The discount rises from 0.5 to 0.9 while rewards and transitions stay fixed. Which zero-based action index wins?

The inputs we are comparing

Next-state probabilities per action

Same in both cases

1	0
0	1

Next-state value estimates

Same in both cases

Immediate expected rewards

Same in both cases

Discount

Immediate reward wins

0.5

Future benefit wins

0.9

Result for immediate reward wins

Index of the largest immediate-plus-future score

For a list use brackets, such as [2, 4]. For a matrix put each row in brackets. Text answers can stay as text.

An immediate reward competes with the destination

This check uses the pieces from “Choose an action after valuing its continuation”.

BOTIE

Botie has three actions and two possible next states. Combine each action's immediate reward with its discounted expected destination value. Return the zero-based index of the highest-valued action; discount is 0.5.

Next-state probabilities per action: 1 0
0.5 0.5
0 1
Next-state value estimates: 02
110
Immediate expected rewards: 05
10
21
Discount: 0.5

What result should your machine produce for this case?

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why is choosing once from supplied values less than a full value-iteration algorithm?

Your notes for later. These stay with this journey.

Planning requires consistent value updates across relevant states, including terminal states.

This is one state's greedy improvement step from supplied values. A full planner must update all relevant states and handle terminals.

Work through the Python challenges →

Continue the courseLearning Action Values From Experience →

← Valuing Rewards That Arrive Later