Advanced Robotics / Reinforcement Learning · Lesson 04
Episodes, Exploration & Exploitation
Est. 70–85 min
Why
Why this lesson matters
In Lesson 3 the robot was given the model (or measured it in a sweep) and computed values by planning. Real robots usually do not know P and R. They must learn from episodes: try, observe the reward and the next state, update, repeat. The simplest method that does this is Q-learning, introduced by Chris Watkins in 1989.
Today you will train Q-learning on the same lab-floor grid from Lesson 3, but the robot never reads the rules — it only sees what happens. You will compare no exploration, a little, a lot, a decaying schedule and “optimistic” starting values, and see one setting that gets completely stuck. On the bench, the light-seeker learns its own Q-table, live.
Concept
Learning from episodes with Q-learning
1. Learning from experience: the Q-learning update
After every step the robot has one transition (s, a, r, s′). Q-learning moves its estimate a small step towards a better guess, called the TD target (temporal-difference target):
target = r if done else r + gamma * max(Q[s2].values()) # reward now + best value next
Q[s][a] += alpha * (target - Q[s][a]) # move a little towards it| Symbol | Name | What it does | Our value |
|---|---|---|---|
| α | learning rate (step size) | 0 = learn nothing, 1 = keep only the newest information | 0.2 (Python), 0.3 (UNO) |
| γ | discount | how much the future counts | 0.9 |
| max Q(s′, ·) | bootstrap | uses its own current estimate of the next state — no need to wait for the episode to end | — |
| Q₀ | initial value | what the table starts with; high values encourage trying everything | 0 (also try +10 and −20) |

2. Exploration vs exploitation
Exploit = pick the action with the highest Q now. Explore = try something else to learn about it. Pure exploiting can lock the robot into the first “okay” habit; pure exploring never uses what it learnt. The simplest compromise is ε-greedy: with probability ε pick a random action, otherwise pick the best. A common schedule starts with ε = 1 (all exploring) and decays it to a small number like 0.05.

3. Episodes: the unit of practice
Each episode starts at reset() and ends when terminated (bench or spill) or truncated (50 steps). We judge learning by the training return per episode (it includes exploration mistakes) and separately by a greedy test (ε = 0, no learning) — just like practice matches and the real match.
Visual
Visualise it: watch Q fill in
A 4 × 4 grid world trained by Q-learning. Watch the Q-values and arrows fill in episode by episode, and change ε to see exploration at work. (Educational simulation.)
Interactive · Grid policy lab · Educational tabular Q-learning
Episodes
0
Last return
0.00
Steps
0
Step episodes with ε-greedy. Watch Q-values / policy arrows update.
Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9
Code lab
Code lab: Q-learning on the lab floor
explore_exploit_lab.py uses the Lesson 3 floor (bench +10, spill −10, −1 per move, 10 % slip). The learner only calls env_step() and sees (s′, r, done). Six settings are trained for 500 episodes with 5 different seeds each, then tested greedily for 1000 episodes.
"""explore_exploit_lab.py - Stage 14 L04: learn from episodes, explore vs exploit.
Same 4 x 4 lab floor as Lesson 3 (S start, G bench +10, X oil spill -10, # pillar,
-1 per move, 10 % slip) - but now the robot is NOT given the model.
It learns Q(s, a) from its own episodes with tabular Q-learning:
Q(s,a) <- Q(s,a) + alpha * (r + gamma * max_a' Q(s',a') - Q(s,a))
and chooses actions epsilon-greedily. We compare exploration settings.
Standard library only.
"""
import random
GRID = ["...G",
".#.X",
"....",
"S..."]
ROWS, COLS = 4, 4
ACTIONS = "URDL"
DELTA = {"U": (-1, 0), "R": (0, 1), "D": (1, 0), "L": (0, -1)}
SIDES = {"U": "LR", "D": "LR", "L": "UD", "R": "UD"}
ARROW = {"U": "^", "R": ">", "D": "v", "L": "<"}
START, SLIP, GAMMA, MAX_STEPS = (3, 0), 0.1, 0.9, 50
def move(s, a):
r, c = s[0] + DELTA[a][0], s[1] + DELTA[a][1]
if 0 <= r < ROWS and 0 <= c < COLS and GRID[r][c] != "#":
return (r, c)
return s
def env_step(s, a, rng):
"""The robot only sees what happens - it never reads this function."""
u = rng.random()
if u >= 1 - SLIP:
a = SIDES[a][0] if u < 1 - SLIP / 2 else SIDES[a][1]
s2 = move(s, a)
ch = GRID[s2[0]][s2[1]]
r = 10.0 if ch == "G" else -10.0 if ch == "X" else -1.0
return s2, r, ch in "GX"
def choose(Q, s, eps, rng):
if rng.random() < eps:
return rng.choice(ACTIONS) # explore
best = max(Q[s].values())
return rng.choice([a for a in ACTIONS if Q[s][a] == best]) # exploit (ties random)
def learn(episodes, eps_fn, alpha=0.2, q0=0.0, seed=0):
rng = random.Random(seed)
states = [(r, c) for r in range(ROWS) for c in range(COLS) if GRID[r][c] != "#"]
Q = {s: {a: q0 for a in ACTIONS} for s in states}
returns, visits = [], {s: 0 for s in states}
for ep in range(episodes):
s, g, eps = START, 0.0, eps_fn(ep)
for _ in range(MAX_STEPS):
a = choose(Q, s, eps, rng)
s2, r, done = env_step(s, a, rng)
target = r if done else r + GAMMA * max(Q[s2].values())
Q[s][a] += alpha * (target - Q[s][a])
visits[s] += 1
g += r
s = s2
if done:
break
returns.append(g)
return Q, returns, visits
def greedy_test(Q, episodes=1000, seed=99):
rng = random.Random(seed)
total, goals = 0.0, 0
for _ in range(episodes):
s, g = START, 0.0
for _ in range(MAX_STEPS):
s, r, done = env_step(s, choose(Q, s, 0.0, rng), rng)
g += r
if done:
goals += GRID[s[0]][s[1]] == "G"
break
total += g
return total / episodes, goals
def show_policy(Q):
for r in range(ROWS):
row = []
for c in range(COLS):
ch = GRID[r][c]
row.append(ch if ch in "#GX" else ARROW[max(ACTIONS, key=lambda a: Q[(r, c)][a])])
print(" " + " ".join(row))
def main():
runs = [
("eps 0.0 + pessimistic Q0=-20", lambda ep: 0.0, -20.0),
("eps 0.0 (pure exploit)", lambda ep: 0.0, 0.0),
("eps 0.1", lambda ep: 0.1, 0.0),
("eps 0.5 (explore a lot)", lambda ep: 0.5, 0.0),
("eps 1.0 -> 0.05 decay", lambda ep: max(0.05, 1.0 - ep / 200), 0.0),
("eps 0.0 + optimistic Q0=+10", lambda ep: 0.0, 10.0),
]
print("Learning curves: average return of the TRAINING episodes (500 episodes, 5 seeds)")
print(f" {'setting':30s} {'ep 1-50':>8s} {'51-100':>8s} {'101-250':>8s} {'251-500':>8s} | greedy test bench/1000")
for name, eps_fn, q0 in runs:
blocks = [0.0] * 4
test_avg, test_goal = 0.0, 0
for seed in range(5):
Q, ret, _ = learn(500, eps_fn, q0=q0, seed=seed)
for i, (a, b) in enumerate([(0, 50), (50, 100), (100, 250), (250, 500)]):
blocks[i] += sum(ret[a:b]) / (b - a) / 5
t, g = greedy_test(Q, seed=99 + seed)
test_avg += t / 5
test_goal += g / 5
print(f" {name:30s} " + " ".join(f"{x:+8.2f}" for x in blocks) + f" | {test_avg:+8.2f} {test_goal:7.0f}")
print("\nOne learner (eps decay, seed 0): greedy policy after 500 episodes")
Q, ret, visits = learn(500, lambda ep: max(0.05, 1.0 - ep / 200), seed=0)
show_policy(Q)
print(f" Q at start: " + " ".join(f"{a}={Q[START][a]:+.2f}" for a in ACTIONS) + " (Lesson 3 V*(start) = 0.97)")
print(" visits per cell (how much each state was explored):")
for r in range(ROWS):
print(" " + " ".join(" ####" if GRID[r][c] == "#" else f"{visits[(r, c)]:5d}" for c in range(COLS)))
print("\nHow many episodes until the training return first averages above 0 (over 20 episodes)?")
for name, eps_fn, q0 in runs:
firsts = []
for seed in range(5):
_, ret, _ = learn(500, eps_fn, q0=q0, seed=seed)
first = next((i + 20 for i in range(len(ret) - 19) if sum(ret[i:i + 20]) / 20 > 0), None)
firsts.append(first)
print(f" {name:30s} {firsts}")
if __name__ == "__main__":
main()Output (run with python3 explore_exploit_lab.py)
Learning curves: average return of the TRAINING episodes (500 episodes, 5 seeds)
setting ep 1-50 51-100 101-250 251-500 | greedy test bench/1000
eps 0.0 + pessimistic Q0=-20 -48.47 -49.06 -49.18 -49.03 | -49.05 29
eps 0.0 (pure exploit) -2.33 +3.20 +3.21 +3.23 | +3.09 944
eps 0.1 -3.52 +2.12 +2.16 +1.80 | -4.07 861
eps 0.5 (explore a lot) -10.94 -8.88 -5.91 -4.38 | +3.38 980
eps 1.0 -> 0.05 decay -27.30 -11.17 +1.35 +3.55 | +4.10 994
eps 0.0 + optimistic Q0=+10 -14.63 +2.66 +3.32 +3.84 | +3.69 983
One learner (eps decay, seed 0): greedy policy after 500 episodes
> > > G
^ # ^ X
^ > ^ v
^ > ^ <
Q at start: U=+1.30 R=+0.08 D=-0.06 L=+0.22 (Lesson 3 V*(start) = 0.97)
visits per cell (how much each state was explored):
602 593 607 0
603 #### 187 0
615 183 210 68
790 236 209 106
How many episodes until the training return first averages above 0 (over 20 episodes)?
eps 0.0 + pessimistic Q0=-20 [None, None, None, None, None]
eps 0.0 (pure exploit) [34, 31, 27, 31, 27]
eps 0.1 [34, 32, 28, 32, 30]
eps 0.5 (explore a lot) [None, 483, None, None, 453]
eps 1.0 -> 0.05 decay [119, 147, 130, 183, 124]
eps 0.0 + optimistic Q0=+10 [63, 55, 64, 65, 66]Try it: run Q-learning on a 4 × 4 grid. Change ε and the number of episodes, and watch the arrows and Q-values fill in. (Educational simulation — not a real RL library.)
Interactive · Grid policy lab · Educational tabular Q-learning
Episodes
0
Last return
0.00
Steps
0
Step episodes with ε-greedy. Watch Q-values / policy arrows update.
Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9
Bench build: the light-seeker learns its own Q-table
q_light_seeker.ino runs real Q-learning on the UNO. State = angle bin (9), actions = LEFT / STAY / RIGHT, reward = +1 brighter, −1 darker, −0.1 per step, +5 and terminated at light level 3. ε decays from 1.0 to 0.05 over 30 episodes; 60 episodes take roughly 5–10 minutes. Every 10 episodes it prints the whole Q-table and the greedy arrow per bin.
| Part | Connection | Note |
|---|---|---|
| SG90/MG90-class servo | signal → D9 | the same light-seeker head |
| Servo supply | red → separate 5–6 V, brown → supply GND | join supply GND to UNO GND |
| LDR + 10 kΩ | LDR 5 V → A0, 10 kΩ A0 → GND | calibrate LEVEL_EDGES first (Lesson 2) |
| Push button | D2 → button → GND | pause / resume — use it before moving the torch |
// q_light_seeker.ino - Stage 14 L04
// Model-free learning on the bench: tabular Q-learning with epsilon-greedy exploration.
// State s = angle bin 0..8 (30..150 deg) Actions: 0 LEFT, 1 STAY, 2 RIGHT (15 deg)
// Reward = -0.1 per step (time), +1 when the light level goes up, -1 when it goes down,
// +5 and TERMINATED when the light level reaches 3 ("found the torch").
// TRUNCATED after 25 steps. Each episode starts at a random bin.
// Epsilon decays from 1.0 to 0.05 over the first 30 episodes.
//
// Wiring: same as L01-L03. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = pause / resume. Serial 115200.
#include <Servo.h>
const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int N = 9, NA = 3, MAX_STEPS = 25, EPISODES = 60;
const int LEVEL_EDGES[3] = {300, 550, 800}; // calibrate for your room (see L02)
const float ALPHA = 0.3, GAMMA = 0.9;
Servo head;
float Q[N][NA]; // 27 floats = 108 bytes, all start at 0
int angleBin = 4, episode = 0;
bool paused = false;
int binToAngle(int b) { return 30 + 15 * b; }
int readLevel() {
long sum = 0;
for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
int raw = sum / 8, level = 0;
while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
return level;
}
void moveToBin(int b) {
int from = head.read(), to = binToAngle(b);
int dir = (to > from) ? 1 : -1;
for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
head.write(to);
delay(300);
}
int bestAction(int s) {
int best = 0;
for (int a = 1; a < NA; a++) if (Q[s][a] > Q[s][best]) best = a;
// break ties at random so equal (untried) actions all get a turn
int ties[NA], n = 0;
for (int a = 0; a < NA; a++) if (Q[s][a] == Q[s][best]) ties[n++] = a;
return ties[random(n)];
}
float maxQ(int s) { return max(Q[s][0], max(Q[s][1], Q[s][2])); }
void checkPause() {
if (digitalRead(BUTTON_PIN) == LOW) {
delay(30);
while (digitalRead(BUTTON_PIN) == LOW) {}
paused = !paused;
Serial.println(paused ? F("# paused") : F("# resumed"));
}
}
void runEpisode() {
float eps = max(0.05, 1.0 - episode / 30.0);
angleBin = random(N);
moveToBin(angleBin);
int level = readLevel();
float g = 0;
int t = 0, explores = 0;
bool terminated = false;
while (t < MAX_STEPS && !terminated) {
int s = angleBin, a;
if (random(1000) < eps * 1000) { a = random(NA); explores++; } // explore
else a = bestAction(s); // exploit
int s2 = constrain(s + (a - 1), 0, N - 1);
if (s2 != s) moveToBin(s2);
int level2 = readLevel();
float r = -0.1;
if (level2 > level) r = 1.0;
else if (level2 < level) r = -1.0;
if (level2 == 3) { r = 5.0; terminated = true; }
float target = terminated ? r : r + GAMMA * maxQ(s2);
Q[s][a] += ALPHA * (target - Q[s][a]); // Q-learning update
angleBin = s2; level = level2; g += r; t++;
}
episode++;
Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
Serial.print(t); Serial.print(','); Serial.print(explores); Serial.print(',');
Serial.print(g, 1); Serial.print(','); Serial.println(terminated ? F("terminated") : F("truncated"));
}
void printPolicy() {
Serial.println(F("# bin: Q[LEFT] Q[STAY] Q[RIGHT] -> greedy"));
for (int s = 0; s < N; s++) {
Serial.print(s); Serial.print(": ");
for (int a = 0; a < NA; a++) { Serial.print(Q[s][a], 2); Serial.print(' '); }
int b = 0;
for (int a = 1; a < NA; a++) if (Q[s][a] > Q[s][b]) b = a;
Serial.println(b == 0 ? F("-> LEFT") : (b == 1 ? F("-> STAY") : F("-> RIGHT")));
}
}
void setup() {
Serial.begin(115200);
pinMode(BUTTON_PIN, INPUT_PULLUP);
randomSeed(analogRead(A5));
head.attach(SERVO_PIN);
head.write(90);
delay(500);
Serial.println(F("episode,epsilon,steps,explores,return,end"));
}
void loop() {
checkPause();
if (paused) return;
if (episode < EPISODES) {
runEpisode();
if (episode % 10 == 0) printPolicy();
}
}episode,epsilon,steps,explores,return,end
1,1.00,<steps>,<explores>,<return>,terminated|truncated
...
# bin: Q[LEFT] Q[STAY] Q[RIGHT] -> greedy
0: <q> <q> <q> -> RIGHT
...Debugging
| Symptom | Likely cause | Fix |
|---|---|---|
| Every episode truncated even late | level 3 never reached in your room | Lower LEVEL_EDGES[2] or bring the torch closer |
| Arrows flip between prints | noisy light readings; α too high | Longer settle delay, steadier torch, try α = 0.1 |
| Python: returns never improve | ε = 0 with pessimistic Q₀, or ties always broken the same way | Add exploration; break ties at random |
| Python: Q-values explode | forgot if done: target = r | Do not bootstrap from terminal states |
| Greedy test much worse than training | some states rarely visited | Train longer, decay ε more slowly, or use optimistic Q₀ |
| UNO resets during learning | servo on UNO 5 V | Separate supply, grounds joined |
Real world
Exploration in the real world

- Boarding a train: the Wikipedia Q-learning article’s own example — rushing in when doors open feels fast, but waiting for people to get off first (found by exploring) gives a shorter total time.
- Robot practice: exploring on real hardware costs time and wear, so labs often explore mostly in simulation and keep ε small on the real robot.
- Safe exploration: on real arms, exploration is limited by joint limits, speed limits and an emergency stop — random actions must still be safe actions.
- Recommendation systems use the same trade-off: show what you know the user likes, or show something new to learn more.

Think
Think about it
Think
The ε = 0.5 learner had the worst training returns in the last block but one of the best greedy tests. How can a robot that “performs badly” have learnt well?
Experiment
Try it in the lab
About 70 minutes. Python 3 for the grid; the light-seeker hardware for live Q-learning.
Experiment
Tune exploration and watch learning
You will need
- `explore_exploit_lab.py` (Python 3, no libraries)
- Grid policy lab (in your browser)
- Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
- LED torch on a stand, spreadsheet software
Steps
- Run `explore_exploit_lab.py` and copy the table. Rank the six settings by greedy test result.
- Change the decay from 200 episodes to 50 and to 400. Which gives the best greedy test? Which learns fastest?
- Change α from 0.2 to 0.05 and to 0.8. What happens to the learning curve and to the noise in Q at start?
- Upload `q_light_seeker.ino`. Save the CSV and plot steps per episode. Note the episode where it starts terminating quickly.
- After episode 60, pause, move the torch to the other side and resume (or reset to retrain). How long does the old policy take to adapt with ε = 0.05?
- In the grid policy lab, compare ε = 0 and ε = 0.3 over the same number of episodes.
Interactive · Grid policy lab · Educational tabular Q-learning
Episodes
0
Last return
0.00
Steps
0
Step episodes with ε-greedy. Watch Q-values / policy arrows update.
Arrows = greedy policy from Q · heat ≈ max Q · G = goal · start bottom-left · α=0.35 γ=0.9
Q-learning turns experience into values one transition at a time. Without exploration it can lock into bad habits; with too much it performs badly while practising. A decaying ε and sensible (or optimistic) starting values usually give the best of both.
Knowledge check
Check your understanding
Knowledge check
1. Q-learning is called “model-free” because…
2. With ε = 0.1, the robot…
3. Q = 2, r = −1, γ = 0.9, maxQ(s′) = 5, α = 0.5. The new Q is…
4. Why did ε = 0 with Q₀ = −20 get stuck?
5. What does a decaying ε schedule do?
6. At a terminal state the TD target should be…
7. Why test with ε = 0 after training?
8. Who introduced Q-learning, and when?
Mini assignment
Hands-on assignment
Mini assignment
Exploration report for your robot
Run, compare and explain exploration settings in simulation and on the bench.
Deliverables
- A table of your Python runs: at least 3 decay lengths and 2 learning rates, with training return (last block) and greedy test result.
- A learning-curve plot (return per 20 episodes) for your best and worst settings.
- The UNO CSV for 60 episodes and a steps-per-episode plot, with the greedy arrow per bin at the end.
- A half-page note: which exploration setting would you use on a real 6-joint arm, and what safety limits would you add?
Do this, step by step
- Run
explore_exploit_lab.pywith its default settings and copy the last-block return. - Try 3 ε-decay lengths and 2 learning rates; fill your results table.
- Plot return per 20 episodes for the best and worst settings.
- Bench: upload
q_light_seeker.ino, run all 60 episodes and save the CSV. - Plot steps per episode and write the final greedy arrow for each bin.
Stuck? Hints, one at a time
- Fast decay means the robot stops exploring early — it may lock onto a poor route. Compare the greedy test, not only training return.
- On the UNO, ε falls from 1.0 to 0.05 over 30 episodes, so early episodes look random on purpose.
- If it never terminates, recalibrate
LEVEL_EDGESso level 3 is reachable with your torch.
Challenges (pick by level)
- 🟢 Beginner: For one week, use ε = 0.25 for your tea or snack stall choice (roll a dice: 1 = try a random stall). Keep a reward score out of 10. Which stall wins?
- 🟡 Intermediate: Add a “softmax” (Boltzmann) exploration option to the Python file and compare it with ε-greedy.
- 🟠 Advanced: Implement SARSA (uses the action actually taken next instead of the max) and compare how close each one walks to the spill during training.
- 🔴 Expert: Save the UNO Q-table to EEPROM after training so it survives a power cycle, and add a “greedy only” mode that does not update Q.
- ⚫ Research: Read the Q-learning article’s section on learning rate and discount. Explain why α = 1 is fine for a deterministic world but bad for our slippery floor.
Next step
So far the reward was simple: +10 at the bench, −1 per move. For a robot arm picking a block, a reward only at the very end makes learning painfully slow. In Lesson 5 — Reward Shaping for Manipulation you will design helpful in-between rewards — and see how a careless one makes the robot cheat.
Look back: Lesson 3: Policies & Value. Next: Lesson 5: Reward Shaping for Manipulation. Related: Stage 13 — Perception, Planning & Manipulation.
Sources
- Wikipedia — Q-learning — model-free, the update rule with α and γ, terminal states, learning rate and discount factor effects, optimistic initial conditions, train-boarding exploration example, introduced by Chris Watkins in 1989.
- Wikipedia — Multi-armed bandit — the exploration–exploitation trade-off.
- Wikipedia — Reinforcement learning — exploration vs exploitation, ε-greedy.
- [Sutton & Barto — Reinforcement Learning: An Introduction, 2nd edition](http://incompleteideas.net/book/the-book-2nd.html) — temporal-difference learning, Q-learning, exploration.
- Gymnasium — Env API — episodes, terminated vs truncated.
- Arduino Servo library (GitHub) — controlling RC hobby servos.
- Photos: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).