Stage 14 · Lesson 04/10

Advanced Robotics / Reinforcement Learning · Lesson 04

Episodes, Exploration & Exploitation

Est. 70–85 min

Why

Why this lesson matters

In Lesson 3 the robot was given the model (or measured it in a sweep) and computed values by planning. Real robots usually do not know P and R. They must learn from episodes: try, observe the reward and the next state, update, repeat. The simplest method that does this is Q-learning, introduced by Chris Watkins in 1989.

Today you will train Q-learning on the same lab-floor grid from Lesson 3, but the robot never reads the rules — it only sees what happens. You will compare no exploration, a little, a lot, a decaying schedule and “optimistic” starting values, and see one setting that gets completely stuck. On the bench, the light-seeker learns its own Q-table, live.

Concept

Learning from episodes with Q-learning

1. Learning from experience: the Q-learning update

After every step the robot has one transition (s, a, r, s′). Q-learning moves its estimate a small step towards a better guess, called the TD target (temporal-difference target):

The Q-learning update (one line of learning) · Python
target = r if done else r + gamma * max(Q[s2].values())   # reward now + best value next
Q[s][a] += alpha * (target - Q[s][a])                       # move a little towards it
It is the Bellman optimality backup from Lesson 3, but using one real sample instead of the full model. For any finite MDP, Q-learning can find an optimal policy (under technical conditions on α and exploration).
SymbolNameWhat it doesOur value
αlearning rate (step size)0 = learn nothing, 1 = keep only the newest information0.2 (Python), 0.3 (UNO)
γdiscounthow much the future counts0.9
max Q(s′, ·)bootstrapuses its own current estimate of the next state — no need to wait for the episode to end—
Q₀initial valuewhat the table starts with; high values encourage trying everything0 (also try +10 and −20)
Two tables of states by actions: the top table is all zeros labelled Initialized, the bottom table has numbers in many cells after training
A Q-table before and after training: rows are states, columns are actions. It starts at zero; every update changes one cell. Our grid has 15 states × 4 actions = 60 cells.Photo: LearnDataSci · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

2. Exploration vs exploitation

Exploit = pick the action with the highest Q now. Explore = try something else to learn about it. Pure exploiting can lock the robot into the first “okay” habit; pure exploring never uses what it learnt. The simplest compromise is ε-greedy: with probability ε pick a random action, otherwise pick the best. A common schedule starts with ε = 1 (all exploring) and decays it to a small number like 0.05.

A long row of brightly lit slot machines in a dark casino
A row of slot machines. Choosing which machine to play when each pays out differently is the classic “multi-armed bandit” — the simplest exploration-vs-exploitation problem (Lesson 1’s light-seeker was one).Photo: User:Yamaguchi先生 · CC BY-SA 3.0 · Source: Wikimedia Commons · resized, converted to WebP

3. Episodes: the unit of practice

Each episode starts at reset() and ends when terminated (bench or spill) or truncated (50 steps). We judge learning by the training return per episode (it includes exploration mistakes) and separately by a greedy test (ε = 0, no learning) — just like practice matches and the real match.

Visual

Visualise it: watch Q fill in

A 4 × 4 grid world trained by Q-learning. Watch the Q-values and arrows fill in episode by episode, and change ε to see exploration at work. (Educational simulation.)

Interactive · Grid policy lab · Educational tabular Q-learning

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / deep RL. Tiny grid + ε-greedy Q-learning for intuition.

Episodes

0

Last return

0.00

Steps

0

↑
↑
↑
G
↑
█
█
↑
↑
█
↑
↑
↑●
↑
↑
↑

Step episodes with ε-greedy. Watch Q-values / policy arrows update.

Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9

Code lab

Code lab: Q-learning on the lab floor

explore_exploit_lab.py uses the Lesson 3 floor (bench +10, spill −10, −1 per move, 10 % slip). The learner only calls env_step() and sees (s′, r, done). Six settings are trained for 500 episodes with 5 different seeds each, then tested greedily for 1000 episodes.

explore_exploit_lab.py · Python
"""explore_exploit_lab.py - Stage 14 L04: learn from episodes, explore vs exploit.

Same 4 x 4 lab floor as Lesson 3 (S start, G bench +10, X oil spill -10, # pillar,
-1 per move, 10 % slip) - but now the robot is NOT given the model.
It learns Q(s, a) from its own episodes with tabular Q-learning:

    Q(s,a) <- Q(s,a) + alpha * (r + gamma * max_a' Q(s',a') - Q(s,a))

and chooses actions epsilon-greedily. We compare exploration settings.
Standard library only.
"""
import random

GRID = ["...G",
        ".#.X",
        "....",
        "S..."]
ROWS, COLS = 4, 4
ACTIONS = "URDL"
DELTA = {"U": (-1, 0), "R": (0, 1), "D": (1, 0), "L": (0, -1)}
SIDES = {"U": "LR", "D": "LR", "L": "UD", "R": "UD"}
ARROW = {"U": "^", "R": ">", "D": "v", "L": "<"}
START, SLIP, GAMMA, MAX_STEPS = (3, 0), 0.1, 0.9, 50


def move(s, a):
    r, c = s[0] + DELTA[a][0], s[1] + DELTA[a][1]
    if 0 <= r < ROWS and 0 <= c < COLS and GRID[r][c] != "#":
        return (r, c)
    return s


def env_step(s, a, rng):
    """The robot only sees what happens - it never reads this function."""
    u = rng.random()
    if u >= 1 - SLIP:
        a = SIDES[a][0] if u < 1 - SLIP / 2 else SIDES[a][1]
    s2 = move(s, a)
    ch = GRID[s2[0]][s2[1]]
    r = 10.0 if ch == "G" else -10.0 if ch == "X" else -1.0
    return s2, r, ch in "GX"


def choose(Q, s, eps, rng):
    if rng.random() < eps:
        return rng.choice(ACTIONS)                       # explore
    best = max(Q[s].values())
    return rng.choice([a for a in ACTIONS if Q[s][a] == best])   # exploit (ties random)


def learn(episodes, eps_fn, alpha=0.2, q0=0.0, seed=0):
    rng = random.Random(seed)
    states = [(r, c) for r in range(ROWS) for c in range(COLS) if GRID[r][c] != "#"]
    Q = {s: {a: q0 for a in ACTIONS} for s in states}
    returns, visits = [], {s: 0 for s in states}
    for ep in range(episodes):
        s, g, eps = START, 0.0, eps_fn(ep)
        for _ in range(MAX_STEPS):
            a = choose(Q, s, eps, rng)
            s2, r, done = env_step(s, a, rng)
            target = r if done else r + GAMMA * max(Q[s2].values())
            Q[s][a] += alpha * (target - Q[s][a])
            visits[s] += 1
            g += r
            s = s2
            if done:
                break
        returns.append(g)
    return Q, returns, visits


def greedy_test(Q, episodes=1000, seed=99):
    rng = random.Random(seed)
    total, goals = 0.0, 0
    for _ in range(episodes):
        s, g = START, 0.0
        for _ in range(MAX_STEPS):
            s, r, done = env_step(s, choose(Q, s, 0.0, rng), rng)
            g += r
            if done:
                goals += GRID[s[0]][s[1]] == "G"
                break
        total += g
    return total / episodes, goals


def show_policy(Q):
    for r in range(ROWS):
        row = []
        for c in range(COLS):
            ch = GRID[r][c]
            row.append(ch if ch in "#GX" else ARROW[max(ACTIONS, key=lambda a: Q[(r, c)][a])])
        print("      " + "  ".join(row))


def main():
    runs = [
        ("eps 0.0 + pessimistic Q0=-20", lambda ep: 0.0, -20.0),
        ("eps 0.0 (pure exploit)", lambda ep: 0.0, 0.0),
        ("eps 0.1", lambda ep: 0.1, 0.0),
        ("eps 0.5 (explore a lot)", lambda ep: 0.5, 0.0),
        ("eps 1.0 -> 0.05 decay", lambda ep: max(0.05, 1.0 - ep / 200), 0.0),
        ("eps 0.0 + optimistic Q0=+10", lambda ep: 0.0, 10.0),
    ]
    print("Learning curves: average return of the TRAINING episodes (500 episodes, 5 seeds)")
    print(f"   {'setting':30s} {'ep 1-50':>8s} {'51-100':>8s} {'101-250':>8s} {'251-500':>8s} | greedy test   bench/1000")
    for name, eps_fn, q0 in runs:
        blocks = [0.0] * 4
        test_avg, test_goal = 0.0, 0
        for seed in range(5):
            Q, ret, _ = learn(500, eps_fn, q0=q0, seed=seed)
            for i, (a, b) in enumerate([(0, 50), (50, 100), (100, 250), (250, 500)]):
                blocks[i] += sum(ret[a:b]) / (b - a) / 5
            t, g = greedy_test(Q, seed=99 + seed)
            test_avg += t / 5
            test_goal += g / 5
        print(f"   {name:30s} " + " ".join(f"{x:+8.2f}" for x in blocks) + f" | {test_avg:+8.2f}   {test_goal:7.0f}")

    print("\nOne learner (eps decay, seed 0): greedy policy after 500 episodes")
    Q, ret, visits = learn(500, lambda ep: max(0.05, 1.0 - ep / 200), seed=0)
    show_policy(Q)
    print(f"   Q at start: " + "  ".join(f"{a}={Q[START][a]:+.2f}" for a in ACTIONS) + "   (Lesson 3 V*(start) = 0.97)")
    print("   visits per cell (how much each state was explored):")
    for r in range(ROWS):
        print("      " + " ".join(" ####" if GRID[r][c] == "#" else f"{visits[(r, c)]:5d}" for c in range(COLS)))

    print("\nHow many episodes until the training return first averages above 0 (over 20 episodes)?")
    for name, eps_fn, q0 in runs:
        firsts = []
        for seed in range(5):
            _, ret, _ = learn(500, eps_fn, q0=q0, seed=seed)
            first = next((i + 20 for i in range(len(ret) - 19) if sum(ret[i:i + 20]) / 20 > 0), None)
            firsts.append(first)
        print(f"   {name:30s} {firsts}")


if __name__ == "__main__":
    main()

Output (run with python3 explore_exploit_lab.py)

Learning curves: average return of the TRAINING episodes (500 episodes, 5 seeds)
   setting                         ep 1-50   51-100  101-250  251-500 | greedy test   bench/1000
   eps 0.0 + pessimistic Q0=-20     -48.47   -49.06   -49.18   -49.03 |   -49.05        29
   eps 0.0 (pure exploit)            -2.33    +3.20    +3.21    +3.23 |    +3.09       944
   eps 0.1                           -3.52    +2.12    +2.16    +1.80 |    -4.07       861
   eps 0.5 (explore a lot)          -10.94    -8.88    -5.91    -4.38 |    +3.38       980
   eps 1.0 -> 0.05 decay            -27.30   -11.17    +1.35    +3.55 |    +4.10       994
   eps 0.0 + optimistic Q0=+10      -14.63    +2.66    +3.32    +3.84 |    +3.69       983

One learner (eps decay, seed 0): greedy policy after 500 episodes
      >  >  >  G
      ^  #  ^  X
      ^  >  ^  v
      ^  >  ^  <
   Q at start: U=+1.30  R=+0.08  D=-0.06  L=+0.22   (Lesson 3 V*(start) = 0.97)
   visits per cell (how much each state was explored):
        602   593   607     0
        603  ####   187     0
        615   183   210    68
        790   236   209   106

How many episodes until the training return first averages above 0 (over 20 episodes)?
   eps 0.0 + pessimistic Q0=-20   [None, None, None, None, None]
   eps 0.0 (pure exploit)         [34, 31, 27, 31, 27]
   eps 0.1                        [34, 32, 28, 32, 30]
   eps 0.5 (explore a lot)        [None, 483, None, None, 453]
   eps 1.0 -> 0.05 decay          [119, 147, 130, 183, 124]
   eps 0.0 + optimistic Q0=+10    [63, 55, 64, 65, 66]

Try it: run Q-learning on a 4 × 4 grid. Change ε and the number of episodes, and watch the arrows and Q-values fill in. (Educational simulation — not a real RL library.)

Interactive · Grid policy lab · Educational tabular Q-learning

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / deep RL. Tiny grid + ε-greedy Q-learning for intuition.

Episodes

0

Last return

0.00

Steps

0

↑
↑
↑
G
↑
█
█
↑
↑
█
↑
↑
↑●
↑
↑
↑

Step episodes with ε-greedy. Watch Q-values / policy arrows update.

Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9

Bench build: the light-seeker learns its own Q-table

q_light_seeker.ino runs real Q-learning on the UNO. State = angle bin (9), actions = LEFT / STAY / RIGHT, reward = +1 brighter, −1 darker, −0.1 per step, +5 and terminated at light level 3. ε decays from 1.0 to 0.05 over 30 episodes; 60 episodes take roughly 5–10 minutes. Every 10 episodes it prints the whole Q-table and the greedy arrow per bin.

Safety: 60 episodes means hundreds of servo moves. Never power the servo from a GPIO pin or the UNO 5 V pin; use its own supply with grounds joined, and stop if the servo gets hot or buzzes at an end stop.
PartConnectionNote
SG90/MG90-class servosignal → D9the same light-seeker head
Servo supplyred → separate 5–6 V, brown → supply GNDjoin supply GND to UNO GND
LDR + 10 kΩLDR 5 V → A0, 10 kΩ A0 → GNDcalibrate LEVEL_EDGES first (Lesson 2)
Push buttonD2 → button → GNDpause / resume — use it before moving the torch
q_light_seeker.ino · Arduino C++
// q_light_seeker.ino - Stage 14 L04
// Model-free learning on the bench: tabular Q-learning with epsilon-greedy exploration.
// State  s = angle bin 0..8 (30..150 deg)      Actions: 0 LEFT, 1 STAY, 2 RIGHT (15 deg)
// Reward = -0.1 per step (time), +1 when the light level goes up, -1 when it goes down,
//          +5 and TERMINATED when the light level reaches 3 ("found the torch").
// TRUNCATED after 25 steps. Each episode starts at a random bin.
// Epsilon decays from 1.0 to 0.05 over the first 30 episodes.
//
// Wiring: same as L01-L03. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = pause / resume. Serial 115200.
#include <Servo.h>

const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int N = 9, NA = 3, MAX_STEPS = 25, EPISODES = 60;
const int LEVEL_EDGES[3] = {300, 550, 800};       // calibrate for your room (see L02)
const float ALPHA = 0.3, GAMMA = 0.9;

Servo head;
float Q[N][NA];                                    // 27 floats = 108 bytes, all start at 0
int angleBin = 4, episode = 0;
bool paused = false;

int binToAngle(int b) { return 30 + 15 * b; }

int readLevel() {
  long sum = 0;
  for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
  int raw = sum / 8, level = 0;
  while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
  return level;
}

void moveToBin(int b) {
  int from = head.read(), to = binToAngle(b);
  int dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
  head.write(to);
  delay(300);
}

int bestAction(int s) {
  int best = 0;
  for (int a = 1; a < NA; a++) if (Q[s][a] > Q[s][best]) best = a;
  // break ties at random so equal (untried) actions all get a turn
  int ties[NA], n = 0;
  for (int a = 0; a < NA; a++) if (Q[s][a] == Q[s][best]) ties[n++] = a;
  return ties[random(n)];
}

float maxQ(int s) { return max(Q[s][0], max(Q[s][1], Q[s][2])); }

void checkPause() {
  if (digitalRead(BUTTON_PIN) == LOW) {
    delay(30);
    while (digitalRead(BUTTON_PIN) == LOW) {}
    paused = !paused;
    Serial.println(paused ? F("# paused") : F("# resumed"));
  }
}

void runEpisode() {
  float eps = max(0.05, 1.0 - episode / 30.0);
  angleBin = random(N);
  moveToBin(angleBin);
  int level = readLevel();
  float g = 0;
  int t = 0, explores = 0;
  bool terminated = false;
  while (t < MAX_STEPS && !terminated) {
    int s = angleBin, a;
    if (random(1000) < eps * 1000) { a = random(NA); explores++; }   // explore
    else a = bestAction(s);                                           // exploit
    int s2 = constrain(s + (a - 1), 0, N - 1);
    if (s2 != s) moveToBin(s2);
    int level2 = readLevel();
    float r = -0.1;
    if (level2 > level) r = 1.0;
    else if (level2 < level) r = -1.0;
    if (level2 == 3) { r = 5.0; terminated = true; }
    float target = terminated ? r : r + GAMMA * maxQ(s2);
    Q[s][a] += ALPHA * (target - Q[s][a]);                            // Q-learning update
    angleBin = s2; level = level2; g += r; t++;
  }
  episode++;
  Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
  Serial.print(t); Serial.print(','); Serial.print(explores); Serial.print(',');
  Serial.print(g, 1); Serial.print(','); Serial.println(terminated ? F("terminated") : F("truncated"));
}

void printPolicy() {
  Serial.println(F("# bin: Q[LEFT] Q[STAY] Q[RIGHT] -> greedy"));
  for (int s = 0; s < N; s++) {
    Serial.print(s); Serial.print(": ");
    for (int a = 0; a < NA; a++) { Serial.print(Q[s][a], 2); Serial.print(' '); }
    int b = 0;
    for (int a = 1; a < NA; a++) if (Q[s][a] > Q[s][b]) b = a;
    Serial.println(b == 0 ? F("-> LEFT") : (b == 1 ? F("-> STAY") : F("-> RIGHT")));
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  randomSeed(analogRead(A5));
  head.attach(SERVO_PIN);
  head.write(90);
  delay(500);
  Serial.println(F("episode,epsilon,steps,explores,return,end"));
}

void loop() {
  checkPause();
  if (paused) return;
  if (episode < EPISODES) {
    runEpisode();
    if (episode % 10 == 0) printPolicy();
  }
}
Serial Monitor (115200) — shape of the output (your numbers will differ) · Text
episode,epsilon,steps,explores,return,end
1,1.00,<steps>,<explores>,<return>,terminated|truncated
...
# bin: Q[LEFT] Q[STAY] Q[RIGHT] -> greedy
0: <q> <q> <q> -> RIGHT
...

Debugging

SymptomLikely causeFix
Every episode truncated even latelevel 3 never reached in your roomLower LEVEL_EDGES[2] or bring the torch closer
Arrows flip between printsnoisy light readings; α too highLonger settle delay, steadier torch, try α = 0.1
Python: returns never improveε = 0 with pessimistic Q₀, or ties always broken the same wayAdd exploration; break ties at random
Python: Q-values explodeforgot if done: target = rDo not bootstrap from terminal states
Greedy test much worse than trainingsome states rarely visitedTrain longer, decay ε more slowly, or use optimistic Q₀
UNO resets during learningservo on UNO 5 VSeparate supply, grounds joined

Real world

Exploration in the real world

A busy roadside pani puri stall in Kolkata with a big bag of puris, steel vessels and customers standing around
A pani puri stall in Kolkata. Going back to your usual stall is exploiting; trying a new one is exploring. Over many evenings, a little exploring is what finds the best one.Photo: Biswarup Ganguly · CC BY 3.0 · Source: Wikimedia Commons · resized, converted to WebP
  • Boarding a train: the Wikipedia Q-learning article’s own example — rushing in when doors open feels fast, but waiting for people to get off first (found by exploring) gives a shorter total time.
  • Robot practice: exploring on real hardware costs time and wear, so labs often explore mostly in simulation and keep ε small on the real robot.
  • Safe exploration: on real arms, exploration is limited by joint limits, speed limits and an emergency stop — random actions must still be safe actions.
  • Recommendation systems use the same trade-off: show what you know the user likes, or show something new to learn more.
NASA’s Curiosity rover on reddish Martian ground, a self-portrait taken by its own arm camera
NASA’s Curiosity rover photographed itself on Mars on Sol 84 (31 October 2012), stitched from 55 images. Real exploration has a cost: every new place visited takes time and energy, so explorers must choose carefully.Photo: NASA/JPL-Caltech/Malin Space Science Systems Derivative work including grading, distortion correction, minor local adjustments and rendering from tiff-file: Julian Herzog · Public domain · Source: Wikimedia Commons · resized, converted to WebP

Think

Think about it

Think

The ε = 0.5 learner had the worst training returns in the last block but one of the best greedy tests. How can a robot that “performs badly” have learnt well?

Experiment

Try it in the lab

About 70 minutes. Python 3 for the grid; the light-seeker hardware for live Q-learning.

Experiment

Tune exploration and watch learning

You will need

  • `explore_exploit_lab.py` (Python 3, no libraries)
  • Grid policy lab (in your browser)
  • Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
  • LED torch on a stand, spreadsheet software

Steps

  1. Run `explore_exploit_lab.py` and copy the table. Rank the six settings by greedy test result.
  2. Change the decay from 200 episodes to 50 and to 400. Which gives the best greedy test? Which learns fastest?
  3. Change α from 0.2 to 0.05 and to 0.8. What happens to the learning curve and to the noise in Q at start?
  4. Upload `q_light_seeker.ino`. Save the CSV and plot steps per episode. Note the episode where it starts terminating quickly.
  5. After episode 60, pause, move the torch to the other side and resume (or reset to retrain). How long does the old policy take to adapt with ε = 0.05?
  6. In the grid policy lab, compare ε = 0 and ε = 0.3 over the same number of episodes.

Interactive · Grid policy lab · Educational tabular Q-learning

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / deep RL. Tiny grid + ε-greedy Q-learning for intuition.

Episodes

0

Last return

0.00

Steps

0

↑
↑
↑
G
↑
█
█
↑
↑
█
↑
↑
↑●
↑
↑
↑

Step episodes with ε-greedy. Watch Q-values / policy arrows update.

Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9

Q-learning turns experience into values one transition at a time. Without exploration it can lock into bad habits; with too much it performs badly while practising. A decaying ε and sensible (or optimistic) starting values usually give the best of both.

Knowledge check

Check your understanding

Knowledge check

1. Q-learning is called “model-free” because…

2. With ε = 0.1, the robot…

3. Q = 2, r = −1, γ = 0.9, maxQ(s′) = 5, α = 0.5. The new Q is…

4. Why did ε = 0 with Q₀ = −20 get stuck?

5. What does a decaying ε schedule do?

6. At a terminal state the TD target should be…

7. Why test with ε = 0 after training?

8. Who introduced Q-learning, and when?

Mini assignment

Hands-on assignment

Mini assignment

Exploration report for your robot

Run, compare and explain exploration settings in simulation and on the bench.

Deliverables

  • A table of your Python runs: at least 3 decay lengths and 2 learning rates, with training return (last block) and greedy test result.
  • A learning-curve plot (return per 20 episodes) for your best and worst settings.
  • The UNO CSV for 60 episodes and a steps-per-episode plot, with the greedy arrow per bin at the end.
  • A half-page note: which exploration setting would you use on a real 6-joint arm, and what safety limits would you add?

Do this, step by step

  1. Run explore_exploit_lab.py with its default settings and copy the last-block return.
  2. Try 3 ε-decay lengths and 2 learning rates; fill your results table.
  3. Plot return per 20 episodes for the best and worst settings.
  4. Bench: upload q_light_seeker.ino, run all 60 episodes and save the CSV.
  5. Plot steps per episode and write the final greedy arrow for each bin.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Fast decay means the robot stops exploring early — it may lock onto a poor route. Compare the greedy test, not only training return.
  2. On the UNO, ε falls from 1.0 to 0.05 over 30 episodes, so early episodes look random on purpose.
  3. If it never terminates, recalibrate LEVEL_EDGES so level 3 is reachable with your torch.

Challenges (pick by level)

  • 🟢 Beginner: For one week, use ε = 0.25 for your tea or snack stall choice (roll a dice: 1 = try a random stall). Keep a reward score out of 10. Which stall wins?
  • 🟡 Intermediate: Add a “softmax” (Boltzmann) exploration option to the Python file and compare it with ε-greedy.
  • 🟠 Advanced: Implement SARSA (uses the action actually taken next instead of the max) and compare how close each one walks to the spill during training.
  • 🔴 Expert: Save the UNO Q-table to EEPROM after training so it survives a power cycle, and add a “greedy only” mode that does not update Q.
  • ⚫ Research: Read the Q-learning article’s section on learning rate and discount. Explain why α = 1 is fine for a deterministic world but bad for our slippery floor.

Next step

So far the reward was simple: +10 at the bench, −1 per move. For a robot arm picking a block, a reward only at the very end makes learning painfully slow. In Lesson 5 — Reward Shaping for Manipulation you will design helpful in-between rewards — and see how a careless one makes the robot cheat.

Look back: Lesson 3: Policies & Value. Next: Lesson 5: Reward Shaping for Manipulation. Related: Stage 13 — Perception, Planning & Manipulation.

Sources

  • Wikipedia — Q-learning — model-free, the update rule with α and γ, terminal states, learning rate and discount factor effects, optimistic initial conditions, train-boarding exploration example, introduced by Chris Watkins in 1989.
  • Wikipedia — Multi-armed bandit — the exploration–exploitation trade-off.
  • Wikipedia — Reinforcement learning — exploration vs exploitation, ε-greedy.
  • [Sutton & Barto — Reinforcement Learning: An Introduction, 2nd edition](http://incompleteideas.net/book/the-book-2nd.html) — temporal-difference learning, Q-learning, exploration.
  • Gymnasium — Env API — episodes, terminated vs truncated.
  • Arduino Servo library (GitHub) — controlling RC hobby servos.
  • Photos: Wikimedia Commons (credits under each image).

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in