← All workshopsRead Reinforcement Learning

Reinforcement Learning

Learn what an action was worth

BOTIE

Build the target from the immediate reward and discounted maximum next-state value.
Ready to test
0 passed · 0 failed · 1 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Build the target from the immediate reward and discounted maximum next-state value.

Let's return to the agent in the maze. It has taken an action, received a reward, and reached a place with several possible next actions. We want to revise the value of the action it just took using both the immediate reward and what may come later.

Inspect the component calculation
Add(
  First contribution: Observed reward
  Second contribution: Multiply(
    Value: Discount
    Multiplier: Largest value(
      Numbers: Next-state action values
    )
  )
)
Examples and results
Observed reward
1
Discount
0.9
Next-state action values
  1. 03
  2. 14
Required result
4.6
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.

A promising route becomes less valuable

This check uses the pieces from “Move an action value toward its observed target”.

BOTIE

Botie expected value 8 from an action, then received reward -2. The available next-action estimates are 3 and 5. Update the old estimate using discount 0.8 and learning rate 0.25.
Current Q estimate
8
Observed reward
-2
Next-state action values
  1. 03
  2. 15
Learning rate
0.25
Discount
0.8

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Which part of the update represents observed evidence, and which part is an estimate about continuation?

One temporal-difference update uses estimated future value; it is not a demonstration of a learned policy.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseTrying an Action Before Knowing Its Value →
← Testing a Forecast as Time Moves Forward