← All workshopsRead Describing What an Action Changes

Reinforcement Learning

Compare the states each action may lead to

BOTIE

Validate the transition rows, then multiply the transition matrix by the state-value vector.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Validate the transition rows, then multiply the transition matrix by the state-value vector.

Botie's watering choice changes the distribution of possible next states. The same state values must be averaged separately for each action.

Validate the transition probability rows, then multiply each row by the aligned state-value vector.

Examples and results
Next-state probabilities per action
0.10.9
0.80.2
Values in next-state order
  1. 00
  2. 110
Required result
  1. 09
  2. 12
Next-state probabilities per action
10
01
Values in next-state order
  1. 03
  2. 17
Required result
  1. 03
  2. 17
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

Transitions and values are supplied. Immediate rewards and a discount are introduced separately before selecting an action. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Three possible places to arrive

This check uses the pieces from “Value the possible outcomes of water and wait”.

BOTIE

Botie can take either of two routes, each ending in one of three states. Rows are routes and columns are destinations, in the same order as the destination values. Calculate each route's expected next-state value.
Next-state probabilities per action
0.50.50
0.10.20.7
Values in next-state order
  1. 00
  2. 14
  3. 210

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What is still missing before these continuation values can choose an action?

Immediate rewards and discounting must join the transition-based continuation before a planning decision.

Transitions and values are supplied. Immediate rewards and a discount are introduced separately before selecting an action.

Work through the Python challenges →
Continue the courseValuing Rewards That Arrive Later →
← Trying an Action Before Knowing Its Value