Learning Action Values From Experience

The planner in the previous lesson had a complete description of the little greenhouse. In a new environment, Botie may only know what happened after an action he tried. He needs a way to learn from that transition without pretending he has already measured every possible outcome.

We can keep one estimate for each state-action pair. After Botie tries an action, its reward gives us new evidence. If the task continues, our estimates for the next state contribute to the target too. We use that target to correct the entry for the action he actually took, and repeat as more experience arrives.

Best next action value = max(4, 6) = 6
Target = 1 + (0.5 × 6) = 4
Difference = 4 − 2 = 2

Old estimate: 2
Update = 0.25 × 2 = 0.5
New estimate = 2 + 0.5 = 2.5

Botie took one action and observed what followed. Compare an ordinary transition with a terminal one, where there is no future action value to add.

2.5

The Q table has one row per state and one column per action. An entry estimates the expected discounted return from taking that action and then following the target behaviour.

The action taken to collect data may be exploratory. Q-learning's target uses a greedy continuation, so it is called off-policy: the data-collection policy and target policy need not be identical.

Q[state, action] → one estimated return

The table size grows with the state and action spaces. A small tabular example is not a solution to arbitrary continuous control.

For a nonterminal transition, add the observed reward to the discounted greatest action value in the next state. Subtract the old entry to obtain a temporal-difference error.

Multiply the error by the learning rate and add it back to the old entry. Compute the target from the old table before mutating it, especially when the next state equals the current state.

target = reward + gamma × max(Q[next_state])
error = target - Q[state,action]
Q_new[state,action] = Q[state,action] + alpha × error

A sampled target can be noisy, so the learning rate controls how much one transition changes the estimate.

If the transition genuinely terminates the task, the target is just its reward. There is no next action whose return should be added.

A time limit imposed by a training script can instead truncate an ongoing process. Whether to bootstrap there depends on the problem definition and whether a valid continuing state is available. Treating all truncations as terminal changes the objective.

True terminal transition: target = reward

Count the final reward once, and do not add an arbitrary value from a terminal table row.

import numpy as np
old_value = 3.0
next_values = np.array([4., 8.])
reward, gamma, rate = 2.0, 0.5, 0.5
target = reward + gamma * next_values.max()
correction = rate * (target - old_value)
print(float(target))
print(float(old_value + correction))

6.0
4.5

Updating only visited pairs requires enough exploration to learn useful alternatives. Tabular convergence results require conditions such as adequate visitation and suitable learning-rate schedules; they do not promise success after a handful of fixed-rate trials.

Inspect learning across several runs, then evaluate the resulting policy without confusing exploratory mistakes with its intended deployment behaviour. The workshop isolates one update, while a replay can repeat transitions to show values changing over experience.

Function approximation, deep Q networks, and safe real-world control introduce additional concerns beyond this tabular lesson.

One table update is not yet a learner. We need something that supplies experience and something that changes its estimates from that experience. In the greenhouse example the environment keeps track of whether a bed is dry or moist. The learner keeps the Q table.

An environment step receives an action and returns a next state, a reward and a flag saying whether the task ended. It does not return the correct action value. The learner must estimate that value by repeating the task. Keep these roles separate in the code so a known environment rule does not accidentally become the answer.

State rows: 0 = dry, 1 = moist
Action columns: 0 = water, 1 = wait
Water: next state moist, reward -1, not terminal
Wait while dry: next state dry, reward -2, not terminal
Wait while moist: reward 3, terminal

These are invented rewards for a tiny task. Reward measures the objective we wrote down, not a universal definition of good behaviour.

At the start of an episode, reset the environment to dry while retaining the learned Q table. Choose an action, receive a transition, update the visited entry and continue until the task ends. This order matters: the next state used in the target is the state returned by the action just taken.

Use a NumPy array of shape (2,2) for Q. Keep state and action as integer indices. A tuple returned by the environment keeps the next state, reward and terminal flag together. A list of episode returns can record how training changes over time without replacing the table.

An epsilon-greedy behaviour policy sometimes chooses an action uniformly at random and otherwise chooses the largest entry in the current row. Early mistakes supply information. The greedy action can change after an update, so choose again at every step.

For each episode:
    state = initial state
    choose action from the current Q row
    next_state, reward, terminal = environment(state, action)
    update only Q[state, action]
    continue from next_state unless terminal

A training step cap prevents an endless toy episode. Reaching that cap is a truncation, not evidence that the environment returned a terminal state.

In the action widget, first water and then wait. The final reward changes the moist/wait entry. Begin another episode and water again. This time the target for the earlier action can use what was learned about the later opportunity. That is how delayed reward moves back through the table.

Run the same experiment in code over many episodes. Inspect the policy, which is the action selected in each state, as well as the numerical entries. A large value for one action is not enough if the competing action has never been tried.

For the greedy two-step route, gamma = 0.9:
Return from moist/wait = 3
Return from dry/water = -1 + 0.9 × 3 = 1.7

The calculation provides an independent check for this deterministic environment. A learned table approaches these values through updates; the numbers are not assigned to it.

Freeze the table and choose greedy actions for evaluation. Do not update Q from those evaluation transitions. Record whether the episode finishes, its reward and its length. For this tiny task, the useful policy waters a dry bed and waits at a moist one.

Repeat training with several seeds. In a stochastic environment, evaluate over multiple episodes too, because one lucky route can hide an unreliable policy. Compare with a simple policy such as always watering; a larger training return alone does not show that learning helped.

A larger state space would make a table impractical. That is the motivation for learning a function that predicts action values. Before introducing a deep Q network, retain the same distinction between experience collection, target calculation and independent evaluation.

The later model changes the representation of values. It does not remove the need to define the environment, explore and evaluate honestly.