Advanced Robotics / Reinforcement Learning · Lesson 03
Policies & Value
Est. 60–75 min
Why
Why this lesson matters
In Lesson 2 we wrote a task as an MDP. Now we ask the two questions every RL method is built around: what should the robot do in each state (the policy)? and how good is each state if it follows that policy (the value)? Once you can compute values, the best policy falls out almost for free.
Today you will compute values by hand-sized code: first for a random policy, then the best possible values with value iteration, and read the best arrows from them. On the bench, the UNO will sweep its light sensor, run value iteration on its own 9-state world and then follow the arrows to the brightest spot.
Concept
Policy, value and the Bellman equation
1. Policy: a rule from state to action
A policy π tells the robot what to do in every state. It can be deterministic (“in cell (2,2) always go UP”) or stochastic (“go UP 80 %, LEFT 20 %”). Our hand-written “careful policy” in Lesson 2 was a deterministic policy written as if statements. A policy table with one arrow per cell is the same thing, just stored as data.

2. Value: how good is this state?
| Name | Symbol | Question it answers |
|---|---|---|
| State value | Vπ(s) | If I am in s and follow π from now on, what return do I expect on average? |
| Action value | Qπ(s, a) | If I am in s, do a first, then follow π, what return do I expect? |
| Optimal value | V*(s), Q*(s, a) | The best expected return any policy can get from s. |
3. The Bellman equation: value = reward now + value next
Richard Bellman’s key idea (dynamic programming) is to break a long problem into one step plus the rest. For a policy π: Vπ(s) = Σ P(s′|s,a) · [R + γ · Vπ(s′)], averaged over the actions π picks. For the best policy, replace the average over actions with a max: **V*(s) = maxₐ Σ P(s′|s,a) · [R + γ · V*(s′)]**. In words: *the value of where I stand = the best of (what I get on the next step + γ × the value of where I land).*
while True:
for s in states: # sweep every state
V[s] = max(q_value(V, s, a) for a in actions) # Bellman optimality backup
if biggest_change < 1e-6:
break
policy = {s: argmax_a q_value(V, s, a) for s in states} # read arrows from V
Visual
Visualise it: arrows on a grid
A 4 × 4 grid world: each cell gets a value estimate and an arrow. Change the settings and watch the arrows point towards the goal. (Educational simulation.)
Interactive · Grid policy lab · Educational tabular Q-learning
Episodes
0
Last return
0.00
Steps
0
Step episodes with ε-greedy. Watch Q-values / policy arrows update.
Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9
Code lab
Code lab: values and arrows on a lab floor
policy_value_lab.py builds a 4 × 4 lab floor with a pillar, the bench G (+10) and an oil spill X (−10). Every move costs −1 and the wheels slip sideways 10 % of the time. It evaluates a random policy, runs value iteration, reads the greedy arrows, prints Q-values for one cell and test-drives both policies for 2000 episodes. Only the standard library is used.
"""policy_value_lab.py - Stage 14 L03: policies and value on a lab-floor grid.
A 4 x 4 lab floor. The robot starts at S (bottom-left) and must deliver a part
to the bench G (top-right, +10, episode ends). Cell X is an oil spill (-10, episode ends).
'#' is a pillar. Every move costs -1. With probability 0.1 the wheels slip and the
robot moves to a random side direction instead (a stochastic MDP).
We compare:
1. V for a RANDOM policy (policy evaluation, Bellman expectation equation)
2. V* and the greedy policy from VALUE ITERATION (Bellman optimality equation)
3. Q(s, a) for one cell, to see what a greedy choice looks at
4. How gamma and the slip probability change the best route
Standard library only.
"""
import random
GRID = ["...G",
".#.X",
"....",
"S..."]
ROWS, COLS = len(GRID), len(GRID[0])
ACTIONS = {"U": (-1, 0), "R": (0, 1), "D": (1, 0), "L": (0, -1)}
ARROW = {"U": "^", "R": ">", "D": "v", "L": "<"}
SIDES = {"U": "LR", "D": "LR", "L": "UD", "R": "UD"}
def cell(r, c):
return GRID[r][c]
STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if cell(r, c) != "#"]
TERMINAL = {s for s in STATES if cell(*s) in "GX"}
def move(s, a):
r, c = s
dr, dc = ACTIONS[a]
nr, nc = r + dr, c + dc
if 0 <= nr < ROWS and 0 <= nc < COLS and cell(nr, nc) != "#":
return (nr, nc)
return s # bumped into a wall or pillar: stay
def reward(s2):
return {"G": 10.0, "X": -10.0}.get(cell(*s2), -1.0)
def transitions(s, a, slip):
"""List of (probability, next_state, reward) - this IS P and R of the MDP."""
out = [(1 - slip, move(s, a))]
for side in SIDES[a]:
out.append((slip / 2, move(s, side)))
return [(p, s2, reward(s2)) for p, s2 in out]
def q_value(V, s, a, gamma, slip):
return sum(p * (r + gamma * (0 if s2 in TERMINAL else V[s2]))
for p, s2, r in transitions(s, a, slip))
def evaluate_random(gamma, slip, sweeps=500):
V = {s: 0.0 for s in STATES}
for _ in range(sweeps):
for s in STATES:
if s not in TERMINAL:
V[s] = sum(0.25 * q_value(V, s, a, gamma, slip) for a in ACTIONS)
return V
def value_iteration(gamma, slip, tol=1e-6):
V = {s: 0.0 for s in STATES}
sweeps = 0
while True:
sweeps += 1
delta = 0.0
for s in STATES:
if s in TERMINAL:
continue
best = max(q_value(V, s, a, gamma, slip) for a in ACTIONS)
delta = max(delta, abs(best - V[s]))
V[s] = best
if delta < tol:
return V, sweeps
def greedy(V, gamma, slip):
return {s: max(ACTIONS, key=lambda a: q_value(V, s, a, gamma, slip))
for s in STATES if s not in TERMINAL}
def show_values(V):
for r in range(ROWS):
row = []
for c in range(COLS):
ch = cell(r, c)
if ch == "#":
row.append(" ####")
elif ch in "GX":
row.append(f" [{ch}] ")
else:
row.append(f"{V[(r, c)]:7.2f}")
print(" " + " ".join(row))
def show_policy(pi):
for r in range(ROWS):
row = []
for c in range(COLS):
ch = cell(r, c)
row.append(ch if ch in "#GX" else ARROW[pi[(r, c)]])
print(" " + " ".join(row))
def run(pi, slip, episodes=2000, seed=7):
rng = random.Random(seed)
total, goals, spills = 0.0, 0, 0
for _ in range(episodes):
s, g = (3, 0), 0.0
for _ in range(50):
a = pi(s, rng)
u = rng.random()
if u < 1 - slip:
s = move(s, a)
else:
s = move(s, SIDES[a][0] if u < 1 - slip / 2 else SIDES[a][1])
g += reward(s)
if s in TERMINAL:
goals += cell(*s) == "G"
spills += cell(*s) == "X"
break
total += g
return total / episodes, goals, spills
def main():
gamma, slip = 0.9, 0.1
print("Lab floor (S start, G bench +10, X oil spill -10, # pillar, -1 per move, 10% slip)")
for row in GRID:
print(" " + " ".join(row))
print("\n1) V(s) for the RANDOM policy (each action 25%), gamma 0.9")
show_values(evaluate_random(gamma, slip))
V, sweeps = value_iteration(gamma, slip)
print(f"\n2) V*(s) from value iteration, gamma 0.9 (converged in {sweeps} sweeps)")
show_values(V)
pi = greedy(V, gamma, slip)
print(" Greedy policy read from V*:")
show_policy(pi)
s = (2, 2)
print(f"\n3) Q(s, a) at cell row 2, col 2 (just below the spill's left neighbour):")
for a in ACTIONS:
print(f" {a} {ARROW[a]} Q = {q_value(V, s, a, gamma, slip):6.2f}")
print(f" greedy choice = {pi[s]} (V* = max of these = {V[s]:.2f})")
print("\n4) Test drive: 2000 episodes with 10% slip")
rand = lambda s, rng: rng.choice("URDL")
best = lambda s, rng: pi[s]
for name, p in [("random policy", rand), ("greedy from V*", best)]:
avg, g, x = run(p, slip)
print(f" {name:15s} avg return {avg:+6.2f} reached bench {g:4d} hit spill {x:4d}")
print("\n5) Change the world, change the best policy")
for gm, sl in [(0.9, 0.0), (0.9, 0.3), (0.5, 0.1)]:
V2, _ = value_iteration(gm, sl)
print(f" gamma {gm}, slip {sl}: V*(start) = {V2[(3, 0)]:6.2f}")
show_policy(greedy(V2, gm, sl))
if __name__ == "__main__":
main()Output (run with python3 policy_value_lab.py)
Lab floor (S start, G bench +10, X oil spill -10, # pillar, -1 per move, 10% slip)
. . . G
. # . X
. . . .
S . . .
1) V(s) for the RANDOM policy (each action 25%), gamma 0.9
-7.68 -5.56 -1.46 [G]
-8.78 #### -7.24 [X]
-9.34 -9.40 -9.04 -9.61
-9.54 -9.54 -9.48 -9.63
2) V*(s) from value iteration, gamma 0.9 (converged in 13 sweeps)
5.47 7.48 9.63 [G]
3.77 #### 6.65 [X]
2.29 2.95 4.63 2.37
0.97 1.56 2.89 1.51
Greedy policy read from V*:
> > > G
^ # ^ X
^ > ^ <
^ ^ ^ <
3) Q(s, a) at cell row 2, col 2 (just below the spill's left neighbour):
U ^ Q = 4.63
R > Q = 1.35
D v Q = 1.58
L < Q = 1.82
greedy choice = U (V* = max of these = 4.63)
4) Test drive: 2000 episodes with 10% slip
random policy avg return -34.85 reached bench 345 hit spill 1080
greedy from V* avg return +4.14 reached bench 1980 hit spill 20
5) Change the world, change the best policy
gamma 0.9, slip 0.0: V*(start) = 1.81
> > > G
^ # ^ X
^ > ^ <
^ ^ ^ ^
gamma 0.9, slip 0.3: V*(start) = -1.66
> > > G
^ # ^ X
^ > ^ <
^ ^ ^ <
gamma 0.5, slip 0.1: V*(start) = -1.73
> > > G
^ # ^ X
^ > ^ <
^ ^ ^ <Try it: a 4 × 4 grid with walls and a goal. Watch the arrows (the greedy policy) appear as the value estimates improve. (Educational simulation — not a real RL library.)
Interactive · Grid policy lab · Educational tabular Q-learning
Episodes
0
Last return
0.00
Steps
0
Step episodes with ε-greedy. Watch Q-values / policy arrows update.
Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9
Bench build: value iteration on the UNO
value_planner.ino turns the light-seeker into a planner. First it sweeps the 9 angle bins and records the light at each — this is its model of the reward. Then it runs value iteration on the 9-state line world (actions LEFT / STAY / RIGHT, small cost for moving, γ = 0.9), prints V and the arrow for every bin, and finally starts from a random bin and follows the arrows until the policy says STAY.
| Part | Connection | Note |
|---|---|---|
| SG90/MG90-class servo | signal → D9 | same light-seeker head as L01/L02 |
| Servo supply | red → separate 5–6 V, brown → supply GND | join supply GND to UNO GND |
| LDR + 10 kΩ | LDR 5 V → A0, 10 kΩ A0 → GND | paper tube on the LDR makes the peak sharper |
| Push button | D2 → button → GND | press after moving the torch: re-sweep and re-plan |
// value_planner.ino - Stage 14 L03
// Plan with value iteration ON the UNO, then follow the greedy policy.
// 1. Sweep: visit the 9 angle bins (30..150 deg), read the light at each -> reward table R[bin].
// 2. Value iteration on the 1-D MDP: states = 9 bins, actions = LEFT / STAY / RIGHT,
// V(s) = max_a [ R(s') - cost(a) + gamma * V(s') ] (s' = s moved by a, walls at the ends)
// 3. Greedy policy: from any start bin, take the action with the highest Q until STAY wins.
//
// Wiring: same as L01/L02. Servo signal D9, servo on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = re-sweep and re-plan. Serial 115200.
#include <Servo.h>
const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int N = 9;
const float GAMMA = 0.9;
const float MOVE_COST = 0.05; // small cost for moving (wear, time); STAY is free
Servo head;
float R[N]; // reward of being at each bin (light, scaled 0..1)
float V[N];
int8_t policyA[N]; // -1 LEFT, 0 STAY, +1 RIGHT
int binToAngle(int b) { return 30 + 15 * b; }
int readLight() {
long sum = 0;
for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
return sum / 8;
}
void moveToBin(int b) {
int from = head.read(), to = binToAngle(b);
int dir = (to > from) ? 1 : -1;
for (int a = from; a != to; a += dir) { head.write(a); delay(10); }
head.write(to);
delay(350);
}
float qValue(int s, int a) {
int s2 = constrain(s + a, 0, N - 1);
float cost = (a == 0) ? 0.0 : MOVE_COST;
return R[s2] - cost + GAMMA * V[s2];
}
void sweepAndPlan() {
Serial.println(F("# sweep: bin,angle,raw,reward"));
for (int b = 0; b < N; b++) {
moveToBin(b);
int raw = readLight();
R[b] = raw / 1023.0;
Serial.print(b); Serial.print(','); Serial.print(binToAngle(b)); Serial.print(',');
Serial.print(raw); Serial.print(','); Serial.println(R[b], 3);
}
for (int s = 0; s < N; s++) V[s] = 0;
int sweeps = 0;
float delta;
do { // value iteration
delta = 0;
for (int s = 0; s < N; s++) {
float best = qValue(s, -1);
for (int a = 0; a <= 1; a++) best = max(best, qValue(s, a));
delta = max(delta, fabs(best - V[s]));
V[s] = best;
}
sweeps++;
} while (delta > 0.0005 && sweeps < 500);
Serial.print(F("# value iteration converged in ")); Serial.print(sweeps); Serial.println(F(" sweeps"));
Serial.println(F("# bin,V,policy"));
for (int s = 0; s < N; s++) {
int bestA = 0;
float bestQ = qValue(s, 0);
for (int a = -1; a <= 1; a += 2) {
float q = qValue(s, a);
if (q > bestQ + 1e-4) { bestQ = q; bestA = a; }
}
policyA[s] = bestA;
Serial.print(s); Serial.print(','); Serial.print(V[s], 2); Serial.print(',');
Serial.println(bestA < 0 ? F("LEFT") : (bestA > 0 ? F("RIGHT") : F("STAY")));
}
}
void followPolicy(int start) {
int s = start;
moveToBin(s);
Serial.print(F("# follow policy from bin ")); Serial.println(s);
for (int step = 0; step < N && policyA[s] != 0; step++) {
s = constrain(s + policyA[s], 0, N - 1);
moveToBin(s);
Serial.print(F(" -> bin ")); Serial.println(s);
}
Serial.print(F("# stopped at bin ")); Serial.print(s);
Serial.print(F(" (angle ")); Serial.print(binToAngle(s)); Serial.println(F(")"));
}
void setup() {
Serial.begin(115200);
pinMode(BUTTON_PIN, INPUT_PULLUP);
randomSeed(analogRead(A5));
head.attach(SERVO_PIN);
head.write(90);
delay(500);
sweepAndPlan();
}
void loop() {
followPolicy(random(N)); // a new random start each time
for (int i = 0; i < 400; i++) { // wait ~4 s; button = move the torch and re-plan
if (digitalRead(BUTTON_PIN) == LOW) { delay(30); sweepAndPlan(); break; }
delay(10);
}
}# sweep: bin,angle,raw,reward
0,30,<raw>,<0..1>
...
8,150,<raw>,<0..1>
# value iteration converged in <n> sweeps
# bin,V,policy
0,<V>,RIGHT
...
<brightest bin>,<V>,STAY
...
# follow policy from bin <random>
-> bin ...
# stopped at bin <brightest> (angle ...)Debugging
| Symptom | Likely cause | Fix |
|---|---|---|
| Every bin says STAY | MOVE_COST bigger than the light differences | Lower MOVE_COST or shade the LDR so the peak is stronger |
| Head stops at a wrong bin | local bright spot (reflection) or reading taken while still moving | Increase settle delay; cover shiny surfaces |
| Value iteration never stops (500 sweeps) | GAMMA set to 1.0 | Keep γ < 1 for a continuing task like this |
| Python values all zero | terminal states included in the update, or forgot to return V | Skip terminal states; check the q_value formula |
| Greedy arrows point into the pillar | move() does not block the pillar | Return the same state when the next cell is a wall or pillar |
| UNO resets when the servo moves | servo on UNO 5 V | Separate supply, grounds joined |
Real world
Policies and value in the real world

- Path planning: grid planners like the ones in Stage 13 compute a cost-to-go for each cell — the same idea as V with costs instead of rewards.
- Games: a board-game player (human or program) judges how good a position is long before the game ends — that judgement is an estimate of value.
- Operations research: MDPs and value iteration came from the 1950s; the Wikipedia history notes that Howard invented policy iteration while optimising Sears catalogue mailing.
- Robot arms: once a value function is learned, the arm’s controller can pick the action with the best Q at every control step.


Think
Think about it
Think
In the Python output, the random policy gives V ≈ −9.5 at the start, but the best policy gives +0.97. The world is identical. Why are the values so different?
Experiment
Try it in the lab
About 60 minutes. Python 3 for the grid; the light-seeker hardware for the planner.
Experiment
Compute, read and follow a value function
You will need
- `policy_value_lab.py` (Python 3, no libraries)
- Grid policy lab (in your browser)
- Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
- LED torch on a stand
Steps
- Run `policy_value_lab.py`. On paper, check one Bellman backup: for cell (0,2) going RIGHT with 10 % slip, compute the Q-value from the printed V* and compare with 9.63.
- Move the spill X to (2,3). Predict how the arrows change, then run and check.
- Set the slip to 0.5. Does the robot ever take a longer route to stay away from the spill?
- Upload `value_planner.ino`. Note which bin it chooses as STAY. Move the torch, press the button and watch it re-plan.
- Change MOVE_COST to 0.2 and re-plan. Do more bins say STAY? Explain using Q = R − cost + γV.
- In the grid policy lab, find a setting where the arrows near a wall look strange, and explain why.
Interactive · Grid policy lab · Educational tabular Q-learning
Episodes
0
Last return
0.00
Steps
0
Step episodes with ε-greedy. Watch Q-values / policy arrows update.
Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9
Values spread out from the reward like heat from a stove: one Bellman sweep at a time. Once values are right, the policy is just “pick the best neighbour”. When costs, slip or γ change, the values change first and the arrows follow.
Knowledge check
Check your understanding
Knowledge check
1. A policy is…
2. Vπ(s) means…
3. The Bellman optimality equation uses which operation over actions?
4. How do you read a greedy policy from V*?
5. Value iteration and policy iteration both need…
6. In the test drive, the greedy policy still hit the spill 20 times. Why?
7. Value iteration is guaranteed to converge when…
8. On the UNO planner, you move the torch but do not press the button. What happens?
Mini assignment
Hands-on assignment
Mini assignment
Value map of your lab
Draw, compute and test a value function for a real route in your lab.
Deliverables
- A grid drawing of a real lab route (at least 4 × 5) with walls, a goal, a hazard and the reward rules.
- Your modified `policy_value_lab.py` with that grid, the printed V* and arrow map, and one Bellman backup checked by hand.
- Results for slip 0.0, 0.1 and 0.3: V*(start) and the test-drive success rate.
- UNO planner log for two torch positions, with the chosen STAY bin and a photo of your setup.
Do this, step by step
- Draw your lab grid (at least 4 × 5) with walls, goal, hazard and rewards.
- Put it into
GRIDinpolicy_value_lab.pyand print V* and the arrow map. - Check one Bellman backup by hand for one cell next to the goal.
- Run with slip 0.0, 0.1 and 0.3; note V*(start) and the test-drive success rate.
- Bench: upload
value_planner.ino, plan for two torch positions, and note the STAY bin each time.
Stuck? Hints, one at a time
- Every row in
GRIDmust have the same length, or the cell lookup breaks. - With
GAMMA0.9 andMOVE_COST0.05, a far bright bin can still lose to a nearer one — that is the policy being sensible, not wrong. - If V does not change after you move the torch, press the button (pin 2) so the sketch sweeps again.
Challenges (pick by level)
- 🟢 Beginner: On a 3 × 3 paper grid with the goal in a corner (+10) and −1 per move, fill in V by hand with γ = 1 and no slip. Then draw the arrows.
- 🟡 Intermediate: Add policy evaluation for your own hand-written policy to the Python file, and compare its V with V*.
- 🟠 Advanced: Implement policy iteration (evaluate, improve, repeat) and count how many improvement rounds it needs versus value-iteration sweeps.
- 🔴 Expert: Give the UNO planner two torches of different brightness. Find a MOVE_COST that makes the head stop at the nearer, dimmer one from some starting bins. Explain with Q-values.
- ⚫ Research: Read the Value iteration and Policy iteration sections of the Wikipedia MDP article. Explain in your own words when policy iteration may be faster.
Next step
Today the robot knew the model (or measured it with a sweep). Real robots usually don’t. In Lesson 4 — Episodes, Exploration & Exploitation the robot learns values from its own trials, and must balance trying new actions with using the best one it knows.
Look back: Lesson 2: States, Actions & Rewards. Next: Lesson 4: Episodes, Exploration & Exploitation. Related: Stage 13 — Perception, Planning & Manipulation.
Sources
- Wikipedia — Markov decision process — policies, value function, Bellman recursion, value iteration (Bellman 1957, converges for γ < 1), policy iteration (Howard 1960, Sears catalogue story), when policy iteration can be faster.
- Wikipedia — Bellman equation — named after Richard E. Bellman; dynamic programming breaks a problem into simpler subproblems; value = payoff from initial choices + value of the remaining problem; “state” as the information needed to decide.
- [Sutton & Barto — Reinforcement Learning: An Introduction, 2nd edition](http://incompleteideas.net/book/the-book-2nd.html) — value functions, Bellman equations, dynamic programming.
- Gymnasium — Create a Custom Environment — grid-world design, boundary handling (stay in place at walls).
- Arduino Servo library (GitHub) — controlling RC hobby servos.
- Photos: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).