Stage 14 · Lesson 03/10

Advanced Robotics / Reinforcement Learning · Lesson 03

Policies & Value

Est. 60–75 min

Why

Why this lesson matters

In Lesson 2 we wrote a task as an MDP. Now we ask the two questions every RL method is built around: what should the robot do in each state (the policy)? and how good is each state if it follows that policy (the value)? Once you can compute values, the best policy falls out almost for free.

Today you will compute values by hand-sized code: first for a random policy, then the best possible values with value iteration, and read the best arrows from them. On the bench, the UNO will sweep its light sensor, run value iteration on its own 9-state world and then follow the arrows to the brightest spot.

Concept

Policy, value and the Bellman equation

1. Policy: a rule from state to action

A policy π tells the robot what to do in every state. It can be deterministic (“in cell (2,2) always go UP”) or stochastic (“go UP 80 %, LEFT 20 %”). Our hand-written “careful policy” in Lesson 2 was a deterministic policy written as if statements. A policy table with one arrow per cell is the same thing, just stored as data.

A green road sign reading BARLACHALA in English and Hindi standing at a high mountain pass with snowy peaks behind
A signboard at Baralacha La, a high pass in the Himalaya. A good sign tells a traveller exactly where they are — that is the state. A policy is like having a direction board at every such point: “from here, go this way”.Photo: John Hill · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

2. Value: how good is this state?

Value is always an expected (average) return, because transitions and rewards can be random. It depends on the policy and on γ.
NameSymbolQuestion it answers
State valueVπ(s)If I am in s and follow π from now on, what return do I expect on average?
Action valueQπ(s, a)If I am in s, do a first, then follow π, what return do I expect?
Optimal valueV*(s), Q*(s, a)The best expected return any policy can get from s.

3. The Bellman equation: value = reward now + value next

Richard Bellman’s key idea (dynamic programming) is to break a long problem into one step plus the rest. For a policy π: Vπ(s) = Σ P(s′|s,a) · [R + γ · Vπ(s′)], averaged over the actions π picks. For the best policy, replace the average over actions with a max: **V*(s) = maxₐ Σ P(s′|s,a) · [R + γ · V*(s′)]**. In words: *the value of where I stand = the best of (what I get on the next step + γ × the value of where I land).*

Value iteration in five lines (the heart of policy_value_lab.py) · Python
while True:
    for s in states:                       # sweep every state
        V[s] = max(q_value(V, s, a) for a in actions)   # Bellman optimality backup
    if biggest_change < 1e-6:
        break
policy = {s: argmax_a q_value(V, s, a) for s in states}  # read arrows from V
A black-and-white line drawing of a trapezoid-shaped hedge maze with a dashed red path from the entrance at the bottom to a dot at the centre
The Hampton Court maze with its solution path. A solution path is one route. A policy is stronger: an arrow in every corridor, so you can reach the centre from wherever you get lost.Photo: ╟─TreasuryTag►hemicycle─╢ · CC BY-SA 3.0 · Source: Wikimedia Commons · resized, converted to WebP

Visual

Visualise it: arrows on a grid

A 4 × 4 grid world: each cell gets a value estimate and an arrow. Change the settings and watch the arrows point towards the goal. (Educational simulation.)

Interactive · Grid policy lab · Educational tabular Q-learning

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / deep RL. Tiny grid + ε-greedy Q-learning for intuition.

Episodes

0

Last return

0.00

Steps

0

↑
↑
↑
G
↑
█
█
↑
↑
█
↑
↑
↑●
↑
↑
↑

Step episodes with ε-greedy. Watch Q-values / policy arrows update.

Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9

Code lab

Code lab: values and arrows on a lab floor

policy_value_lab.py builds a 4 × 4 lab floor with a pillar, the bench G (+10) and an oil spill X (−10). Every move costs −1 and the wheels slip sideways 10 % of the time. It evaluates a random policy, runs value iteration, reads the greedy arrows, prints Q-values for one cell and test-drives both policies for 2000 episodes. Only the standard library is used.

policy_value_lab.py · Python
"""policy_value_lab.py - Stage 14 L03: policies and value on a lab-floor grid.

A 4 x 4 lab floor. The robot starts at S (bottom-left) and must deliver a part
to the bench G (top-right, +10, episode ends). Cell X is an oil spill (-10, episode ends).
'#' is a pillar. Every move costs -1. With probability 0.1 the wheels slip and the
robot moves to a random side direction instead (a stochastic MDP).

We compare:
  1. V for a RANDOM policy (policy evaluation, Bellman expectation equation)
  2. V* and the greedy policy from VALUE ITERATION (Bellman optimality equation)
  3. Q(s, a) for one cell, to see what a greedy choice looks at
  4. How gamma and the slip probability change the best route
Standard library only.
"""
import random

GRID = ["...G",
        ".#.X",
        "....",
        "S..."]
ROWS, COLS = len(GRID), len(GRID[0])
ACTIONS = {"U": (-1, 0), "R": (0, 1), "D": (1, 0), "L": (0, -1)}
ARROW = {"U": "^", "R": ">", "D": "v", "L": "<"}
SIDES = {"U": "LR", "D": "LR", "L": "UD", "R": "UD"}


def cell(r, c):
    return GRID[r][c]


STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if cell(r, c) != "#"]
TERMINAL = {s for s in STATES if cell(*s) in "GX"}


def move(s, a):
    r, c = s
    dr, dc = ACTIONS[a]
    nr, nc = r + dr, c + dc
    if 0 <= nr < ROWS and 0 <= nc < COLS and cell(nr, nc) != "#":
        return (nr, nc)
    return s  # bumped into a wall or pillar: stay


def reward(s2):
    return {"G": 10.0, "X": -10.0}.get(cell(*s2), -1.0)


def transitions(s, a, slip):
    """List of (probability, next_state, reward) - this IS P and R of the MDP."""
    out = [(1 - slip, move(s, a))]
    for side in SIDES[a]:
        out.append((slip / 2, move(s, side)))
    return [(p, s2, reward(s2)) for p, s2 in out]


def q_value(V, s, a, gamma, slip):
    return sum(p * (r + gamma * (0 if s2 in TERMINAL else V[s2]))
               for p, s2, r in transitions(s, a, slip))


def evaluate_random(gamma, slip, sweeps=500):
    V = {s: 0.0 for s in STATES}
    for _ in range(sweeps):
        for s in STATES:
            if s not in TERMINAL:
                V[s] = sum(0.25 * q_value(V, s, a, gamma, slip) for a in ACTIONS)
    return V


def value_iteration(gamma, slip, tol=1e-6):
    V = {s: 0.0 for s in STATES}
    sweeps = 0
    while True:
        sweeps += 1
        delta = 0.0
        for s in STATES:
            if s in TERMINAL:
                continue
            best = max(q_value(V, s, a, gamma, slip) for a in ACTIONS)
            delta = max(delta, abs(best - V[s]))
            V[s] = best
        if delta < tol:
            return V, sweeps


def greedy(V, gamma, slip):
    return {s: max(ACTIONS, key=lambda a: q_value(V, s, a, gamma, slip))
            for s in STATES if s not in TERMINAL}


def show_values(V):
    for r in range(ROWS):
        row = []
        for c in range(COLS):
            ch = cell(r, c)
            if ch == "#":
                row.append("   ####")
            elif ch in "GX":
                row.append(f"   [{ch}] ")
            else:
                row.append(f"{V[(r, c)]:7.2f}")
        print("   " + " ".join(row))


def show_policy(pi):
    for r in range(ROWS):
        row = []
        for c in range(COLS):
            ch = cell(r, c)
            row.append(ch if ch in "#GX" else ARROW[pi[(r, c)]])
        print("      " + "  ".join(row))


def run(pi, slip, episodes=2000, seed=7):
    rng = random.Random(seed)
    total, goals, spills = 0.0, 0, 0
    for _ in range(episodes):
        s, g = (3, 0), 0.0
        for _ in range(50):
            a = pi(s, rng)
            u = rng.random()
            if u < 1 - slip:
                s = move(s, a)
            else:
                s = move(s, SIDES[a][0] if u < 1 - slip / 2 else SIDES[a][1])
            g += reward(s)
            if s in TERMINAL:
                goals += cell(*s) == "G"
                spills += cell(*s) == "X"
                break
        total += g
    return total / episodes, goals, spills


def main():
    gamma, slip = 0.9, 0.1
    print("Lab floor (S start, G bench +10, X oil spill -10, # pillar, -1 per move, 10% slip)")
    for row in GRID:
        print("      " + "  ".join(row))

    print("\n1) V(s) for the RANDOM policy (each action 25%), gamma 0.9")
    show_values(evaluate_random(gamma, slip))

    V, sweeps = value_iteration(gamma, slip)
    print(f"\n2) V*(s) from value iteration, gamma 0.9 (converged in {sweeps} sweeps)")
    show_values(V)
    pi = greedy(V, gamma, slip)
    print("   Greedy policy read from V*:")
    show_policy(pi)

    s = (2, 2)
    print(f"\n3) Q(s, a) at cell row 2, col 2 (just below the spill's left neighbour):")
    for a in ACTIONS:
        print(f"   {a} {ARROW[a]}  Q = {q_value(V, s, a, gamma, slip):6.2f}")
    print(f"   greedy choice = {pi[s]}  (V* = max of these = {V[s]:.2f})")

    print("\n4) Test drive: 2000 episodes with 10% slip")
    rand = lambda s, rng: rng.choice("URDL")
    best = lambda s, rng: pi[s]
    for name, p in [("random policy", rand), ("greedy from V*", best)]:
        avg, g, x = run(p, slip)
        print(f"   {name:15s} avg return {avg:+6.2f}  reached bench {g:4d}  hit spill {x:4d}")

    print("\n5) Change the world, change the best policy")
    for gm, sl in [(0.9, 0.0), (0.9, 0.3), (0.5, 0.1)]:
        V2, _ = value_iteration(gm, sl)
        print(f"   gamma {gm}, slip {sl}:  V*(start) = {V2[(3, 0)]:6.2f}")
        show_policy(greedy(V2, gm, sl))


if __name__ == "__main__":
    main()

Output (run with python3 policy_value_lab.py)

Lab floor (S start, G bench +10, X oil spill -10, # pillar, -1 per move, 10% slip)
      .  .  .  G
      .  #  .  X
      .  .  .  .
      S  .  .  .

1) V(s) for the RANDOM policy (each action 25%), gamma 0.9
     -7.68   -5.56   -1.46    [G] 
     -8.78    ####   -7.24    [X] 
     -9.34   -9.40   -9.04   -9.61
     -9.54   -9.54   -9.48   -9.63

2) V*(s) from value iteration, gamma 0.9 (converged in 13 sweeps)
      5.47    7.48    9.63    [G] 
      3.77    ####    6.65    [X] 
      2.29    2.95    4.63    2.37
      0.97    1.56    2.89    1.51
   Greedy policy read from V*:
      >  >  >  G
      ^  #  ^  X
      ^  >  ^  <
      ^  ^  ^  <

3) Q(s, a) at cell row 2, col 2 (just below the spill's left neighbour):
   U ^  Q =   4.63
   R >  Q =   1.35
   D v  Q =   1.58
   L <  Q =   1.82
   greedy choice = U  (V* = max of these = 4.63)

4) Test drive: 2000 episodes with 10% slip
   random policy   avg return -34.85  reached bench  345  hit spill 1080
   greedy from V*  avg return  +4.14  reached bench 1980  hit spill   20

5) Change the world, change the best policy
   gamma 0.9, slip 0.0:  V*(start) =   1.81
      >  >  >  G
      ^  #  ^  X
      ^  >  ^  <
      ^  ^  ^  ^
   gamma 0.9, slip 0.3:  V*(start) =  -1.66
      >  >  >  G
      ^  #  ^  X
      ^  >  ^  <
      ^  ^  ^  <
   gamma 0.5, slip 0.1:  V*(start) =  -1.73
      >  >  >  G
      ^  #  ^  X
      ^  >  ^  <
      ^  ^  ^  <

Try it: a 4 × 4 grid with walls and a goal. Watch the arrows (the greedy policy) appear as the value estimates improve. (Educational simulation — not a real RL library.)

Interactive · Grid policy lab · Educational tabular Q-learning

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / deep RL. Tiny grid + ε-greedy Q-learning for intuition.

Episodes

0

Last return

0.00

Steps

0

↑
↑
↑
G
↑
█
█
↑
↑
█
↑
↑
↑●
↑
↑
↑

Step episodes with ε-greedy. Watch Q-values / policy arrows update.

Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9

Bench build: value iteration on the UNO

value_planner.ino turns the light-seeker into a planner. First it sweeps the 9 angle bins and records the light at each — this is its model of the reward. Then it runs value iteration on the 9-state line world (actions LEFT / STAY / RIGHT, small cost for moving, γ = 0.9), prints V and the arrow for every bin, and finally starts from a random bin and follows the arrows until the policy says STAY.

Safety: the servo must not be powered from a GPIO pin or the UNO 5 V pin — use a separate supply with grounds joined. Mount the torch on a stand so it cannot fall onto the circuit.
PartConnectionNote
SG90/MG90-class servosignal → D9same light-seeker head as L01/L02
Servo supplyred → separate 5–6 V, brown → supply GNDjoin supply GND to UNO GND
LDR + 10 kΩLDR 5 V → A0, 10 kΩ A0 → GNDpaper tube on the LDR makes the peak sharper
Push buttonD2 → button → GNDpress after moving the torch: re-sweep and re-plan
value_planner.ino · Arduino C++
// value_planner.ino - Stage 14 L03
// Plan with value iteration ON the UNO, then follow the greedy policy.
// 1. Sweep: visit the 9 angle bins (30..150 deg), read the light at each -> reward table R[bin].
// 2. Value iteration on the 1-D MDP: states = 9 bins, actions = LEFT / STAY / RIGHT,
//    V(s) = max_a [ R(s') - cost(a) + gamma * V(s') ]   (s' = s moved by a, walls at the ends)
// 3. Greedy policy: from any start bin, take the action with the highest Q until STAY wins.
//
// Wiring: same as L01/L02. Servo signal D9, servo on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = re-sweep and re-plan. Serial 115200.
#include <Servo.h>

const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int N = 9;
const float GAMMA = 0.9;
const float MOVE_COST = 0.05;     // small cost for moving (wear, time); STAY is free

Servo head;
float R[N];                        // reward of being at each bin (light, scaled 0..1)
float V[N];
int8_t policyA[N];                 // -1 LEFT, 0 STAY, +1 RIGHT

int binToAngle(int b) { return 30 + 15 * b; }

int readLight() {
  long sum = 0;
  for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
  return sum / 8;
}

void moveToBin(int b) {
  int from = head.read(), to = binToAngle(b);
  int dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { head.write(a); delay(10); }
  head.write(to);
  delay(350);
}

float qValue(int s, int a) {
  int s2 = constrain(s + a, 0, N - 1);
  float cost = (a == 0) ? 0.0 : MOVE_COST;
  return R[s2] - cost + GAMMA * V[s2];
}

void sweepAndPlan() {
  Serial.println(F("# sweep: bin,angle,raw,reward"));
  for (int b = 0; b < N; b++) {
    moveToBin(b);
    int raw = readLight();
    R[b] = raw / 1023.0;
    Serial.print(b); Serial.print(','); Serial.print(binToAngle(b)); Serial.print(',');
    Serial.print(raw); Serial.print(','); Serial.println(R[b], 3);
  }
  for (int s = 0; s < N; s++) V[s] = 0;
  int sweeps = 0;
  float delta;
  do {                                   // value iteration
    delta = 0;
    for (int s = 0; s < N; s++) {
      float best = qValue(s, -1);
      for (int a = 0; a <= 1; a++) best = max(best, qValue(s, a));
      delta = max(delta, fabs(best - V[s]));
      V[s] = best;
    }
    sweeps++;
  } while (delta > 0.0005 && sweeps < 500);

  Serial.print(F("# value iteration converged in ")); Serial.print(sweeps); Serial.println(F(" sweeps"));
  Serial.println(F("# bin,V,policy"));
  for (int s = 0; s < N; s++) {
    int bestA = 0;
    float bestQ = qValue(s, 0);
    for (int a = -1; a <= 1; a += 2) {
      float q = qValue(s, a);
      if (q > bestQ + 1e-4) { bestQ = q; bestA = a; }
    }
    policyA[s] = bestA;
    Serial.print(s); Serial.print(','); Serial.print(V[s], 2); Serial.print(',');
    Serial.println(bestA < 0 ? F("LEFT") : (bestA > 0 ? F("RIGHT") : F("STAY")));
  }
}

void followPolicy(int start) {
  int s = start;
  moveToBin(s);
  Serial.print(F("# follow policy from bin ")); Serial.println(s);
  for (int step = 0; step < N && policyA[s] != 0; step++) {
    s = constrain(s + policyA[s], 0, N - 1);
    moveToBin(s);
    Serial.print(F("  -> bin ")); Serial.println(s);
  }
  Serial.print(F("# stopped at bin ")); Serial.print(s);
  Serial.print(F(" (angle ")); Serial.print(binToAngle(s)); Serial.println(F(")"));
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  randomSeed(analogRead(A5));
  head.attach(SERVO_PIN);
  head.write(90);
  delay(500);
  sweepAndPlan();
}

void loop() {
  followPolicy(random(N));               // a new random start each time
  for (int i = 0; i < 400; i++) {        // wait ~4 s; button = move the torch and re-plan
    if (digitalRead(BUTTON_PIN) == LOW) { delay(30); sweepAndPlan(); break; }
    delay(10);
  }
}
Serial Monitor (115200) — shape of the output (your numbers depend on the torch and room) · Text
# sweep: bin,angle,raw,reward
0,30,<raw>,<0..1>
...
8,150,<raw>,<0..1>
# value iteration converged in <n> sweeps
# bin,V,policy
0,<V>,RIGHT
...
<brightest bin>,<V>,STAY
...
# follow policy from bin <random>
  -> bin ...
# stopped at bin <brightest> (angle ...)

Debugging

SymptomLikely causeFix
Every bin says STAYMOVE_COST bigger than the light differencesLower MOVE_COST or shade the LDR so the peak is stronger
Head stops at a wrong binlocal bright spot (reflection) or reading taken while still movingIncrease settle delay; cover shiny surfaces
Value iteration never stops (500 sweeps)GAMMA set to 1.0Keep γ < 1 for a continuing task like this
Python values all zeroterminal states included in the update, or forgot to return VSkip terminal states; check the q_value formula
Greedy arrows point into the pillarmove() does not block the pillarReturn the same state when the next cell is a wall or pillar
UNO resets when the servo movesservo on UNO 5 VSeparate supply, grounds joined

Real world

Policies and value in the real world

A round black robot vacuum cleaner docked at its charging station against a white wall
A robot vacuum returning to its dock. “Go back to the charger” is a policy over states like position and battery level; a good robot effectively knows that being far from the dock with a low battery is a low-value state.Photo: ajay_suresh · CC BY 2.0 · Source: Wikimedia Commons · resized, converted to WebP
  • Path planning: grid planners like the ones in Stage 13 compute a cost-to-go for each cell — the same idea as V with costs instead of rewards.
  • Games: a board-game player (human or program) judges how good a position is long before the game ends — that judgement is an estimate of value.
  • Operations research: MDPs and value iteration came from the 1950s; the Wikipedia history notes that Howard invented policy iteration while optimising Sears catalogue mailing.
  • Robot arms: once a value function is learned, the arm’s controller can pick the action with the best Q at every control step.
A wooden chessboard with a full set of Staunton-style wooden chess pieces in the starting position
A chessboard at the start of a game. A strong player looks at a position and feels “this is good for me” without seeing the end — an estimate of value.Photo: Wilfredor · CC0 · Source: Wikimedia Commons · resized, converted to WebP
A narrow path between tall green hedges in a garden maze on a sunny day
Inside the Hampton Court hedge maze. Inside, you only see the hedges near you — a robot also only sees its current state. A policy with an arrow for every spot is what lets it find the way.Photo: Kris Arnold · CC BY-SA 2.0 · Source: Wikimedia Commons · resized, converted to WebP

Think

Think about it

Think

In the Python output, the random policy gives V ≈ −9.5 at the start, but the best policy gives +0.97. The world is identical. Why are the values so different?

Experiment

Try it in the lab

About 60 minutes. Python 3 for the grid; the light-seeker hardware for the planner.

Experiment

Compute, read and follow a value function

You will need

  • `policy_value_lab.py` (Python 3, no libraries)
  • Grid policy lab (in your browser)
  • Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
  • LED torch on a stand

Steps

  1. Run `policy_value_lab.py`. On paper, check one Bellman backup: for cell (0,2) going RIGHT with 10 % slip, compute the Q-value from the printed V* and compare with 9.63.
  2. Move the spill X to (2,3). Predict how the arrows change, then run and check.
  3. Set the slip to 0.5. Does the robot ever take a longer route to stay away from the spill?
  4. Upload `value_planner.ino`. Note which bin it chooses as STAY. Move the torch, press the button and watch it re-plan.
  5. Change MOVE_COST to 0.2 and re-plan. Do more bins say STAY? Explain using Q = R − cost + γV.
  6. In the grid policy lab, find a setting where the arrows near a wall look strange, and explain why.

Interactive · Grid policy lab · Educational tabular Q-learning

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / deep RL. Tiny grid + ε-greedy Q-learning for intuition.

Episodes

0

Last return

0.00

Steps

0

↑
↑
↑
G
↑
█
█
↑
↑
█
↑
↑
↑●
↑
↑
↑

Step episodes with ε-greedy. Watch Q-values / policy arrows update.

Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9

Values spread out from the reward like heat from a stove: one Bellman sweep at a time. Once values are right, the policy is just “pick the best neighbour”. When costs, slip or γ change, the values change first and the arrows follow.

Knowledge check

Check your understanding

Knowledge check

1. A policy is…

2. Vπ(s) means…

3. The Bellman optimality equation uses which operation over actions?

4. How do you read a greedy policy from V*?

5. Value iteration and policy iteration both need…

6. In the test drive, the greedy policy still hit the spill 20 times. Why?

7. Value iteration is guaranteed to converge when…

8. On the UNO planner, you move the torch but do not press the button. What happens?

Mini assignment

Hands-on assignment

Mini assignment

Value map of your lab

Draw, compute and test a value function for a real route in your lab.

Deliverables

  • A grid drawing of a real lab route (at least 4 × 5) with walls, a goal, a hazard and the reward rules.
  • Your modified `policy_value_lab.py` with that grid, the printed V* and arrow map, and one Bellman backup checked by hand.
  • Results for slip 0.0, 0.1 and 0.3: V*(start) and the test-drive success rate.
  • UNO planner log for two torch positions, with the chosen STAY bin and a photo of your setup.

Do this, step by step

  1. Draw your lab grid (at least 4 × 5) with walls, goal, hazard and rewards.
  2. Put it into GRID in policy_value_lab.py and print V* and the arrow map.
  3. Check one Bellman backup by hand for one cell next to the goal.
  4. Run with slip 0.0, 0.1 and 0.3; note V*(start) and the test-drive success rate.
  5. Bench: upload value_planner.ino, plan for two torch positions, and note the STAY bin each time.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Every row in GRID must have the same length, or the cell lookup breaks.
  2. With GAMMA 0.9 and MOVE_COST 0.05, a far bright bin can still lose to a nearer one — that is the policy being sensible, not wrong.
  3. If V does not change after you move the torch, press the button (pin 2) so the sketch sweeps again.

Challenges (pick by level)

  • 🟢 Beginner: On a 3 × 3 paper grid with the goal in a corner (+10) and −1 per move, fill in V by hand with γ = 1 and no slip. Then draw the arrows.
  • 🟡 Intermediate: Add policy evaluation for your own hand-written policy to the Python file, and compare its V with V*.
  • 🟠 Advanced: Implement policy iteration (evaluate, improve, repeat) and count how many improvement rounds it needs versus value-iteration sweeps.
  • 🔴 Expert: Give the UNO planner two torches of different brightness. Find a MOVE_COST that makes the head stop at the nearer, dimmer one from some starting bins. Explain with Q-values.
  • ⚫ Research: Read the Value iteration and Policy iteration sections of the Wikipedia MDP article. Explain in your own words when policy iteration may be faster.

Next step

Today the robot knew the model (or measured it with a sweep). Real robots usually don’t. In Lesson 4 — Episodes, Exploration & Exploitation the robot learns values from its own trials, and must balance trying new actions with using the best one it knows.

Look back: Lesson 2: States, Actions & Rewards. Next: Lesson 4: Episodes, Exploration & Exploitation. Related: Stage 13 — Perception, Planning & Manipulation.

Sources

  • Wikipedia — Markov decision process — policies, value function, Bellman recursion, value iteration (Bellman 1957, converges for γ < 1), policy iteration (Howard 1960, Sears catalogue story), when policy iteration can be faster.
  • Wikipedia — Bellman equation — named after Richard E. Bellman; dynamic programming breaks a problem into simpler subproblems; value = payoff from initial choices + value of the remaining problem; “state” as the information needed to decide.
  • [Sutton & Barto — Reinforcement Learning: An Introduction, 2nd edition](http://incompleteideas.net/book/the-book-2nd.html) — value functions, Bellman equations, dynamic programming.
  • Gymnasium — Create a Custom Environment — grid-world design, boundary handling (stay in place at walls).
  • Arduino Servo library (GitHub) — controlling RC hobby servos.
  • Photos: Wikimedia Commons (credits under each image).

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in