Advanced Robotics / Reinforcement Learning · Lesson 10
DQN Intuition Challenge
Est. 75–95 min
Why
Why this challenge matters
Lessons 6–9 gave you the pieces one by one: why a Q-table explodes, how a function can generalise, how experience replay and a target network make DQN stable, and why max() overestimates. This lesson is a challenge. You get one broken DQN with four classic mistakes, a scoreboard, and a real two-servo robot. Your job is to fix it step by step, measure every step honestly over several seeds, and explain *why* each fix helped — or did not.
This is exactly how robot-learning engineers work. Nobody writes a perfect learner in one go. They change one thing at a time, keep a log, and trust numbers from several seeds rather than one lucky run.
Concept
The challenge brief: fix a broken DQN
1. The robot: a pan-tilt light-seeker
Two hobby servos make a “head”: one turns left-right (pan), one tilts up-down. The state is just two angles, (pan, tilt), each 0–180°. In the simulation the lamp sits at (120°, 60°). There are 5 actions: pan −10°, pan +10°, tilt −10°, tilt +10°, stay. Each step the reward is −(distance to lamp) ÷ 50, for 30 steps. Success = ending within 15° of the lamp.

Why not a table? With 10° bins you need 19 × 19 positions × 5 actions = 1805 values, each learnt separately, and the robot still cannot tell 61° from 69°. A tiny network (2 inputs → 24 tanh → 5 Q-values) shares what it learns between nearby angles — the Lesson 7 idea.
2. The four classic mistakes in BROKEN
| Mistake | Why it hurts | Fix (lesson) |
|---|---|---|
| Raw inputs 0–180 | tanh units saturate; gradients are tiny or huge | scale to −1…1 (L7) |
| Learning rate 0.05 | big jumps; values swing | smaller step, 0.01 (L9) |
| Learn only from the last step | consecutive steps are almost the same, so the network over-fits the latest moment | experience replay, mini-batch of 8 (L8) |
| Target from the same network | the goalpost moves on every update | target network synced every 500 updates (L8) |
| (bonus) plain max target | max of noisy Q-values is too high | Double DQN target (L9) |
BROKEN = dict(scale_inputs=False, replay=False, target=False, double=False, lr=0.05)
FIXED = dict(scale_inputs=True, replay=True, target=True, double=True, lr=0.01)3. The rules of the challenge
- One change at a time. If you change two things and it gets better, you do not know which one helped.
- Several seeds. A tiny DQN is fragile. One seed can lie — in both directions.
- Test greedily. Training return includes exploration. Score the learnt policy with ε = 0 from fresh random starts (the script uses 40).
- Write down surprises. A fix that makes things worse is data, not failure.

Visual
Visualise it: the whole DQN loop in one place
Everything from Lessons 6–9 in one picture: a small Q-network, a replay buffer and a lagged target network you can sync by hand. (Educational simulation.)
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Code lab
Code lab: the scoreboard
dqn_challenge.py trains the BROKEN config, then switches the fixes on cumulatively in this order: scale inputs → lower learning rate → replay (buffer 4000, batch 8) → target network (sync every 500 updates) → Double DQN target. Each config runs 120 episodes on 3 seeds and is scored greedily from 40 random starts. Standard library only; it took about 9 seconds on our machine.
"""dqn_challenge.py - Stage 14 L10: the DQN intuition challenge.
Task: a pan-tilt light-seeker. State = (pan, tilt) in degrees, 0..180 each.
The lamp is at (120, 60). Actions: pan -10/+10, tilt -10/+10, stay (5 actions).
Reward = -distance_to_lamp / 50 each step; 30 steps per episode, no early stop.
Success = ending within 15 deg of the lamp.
A Q-table with 10 deg bins would need 19 x 19 x 5 = 1805 values, each learnt separately.
Instead we use a tiny DQN (2 inputs -> 24 tanh -> 5). The BROKEN config below has
four classic mistakes. Fix them one at a time and watch the scoreboard.
Standard library only (about 10 seconds).
"""
import math
import random
H, NA, GAMMA = 24, 5, 0.9
LAMP = (120.0, 60.0)
MOVES = [(-10, 0), (10, 0), (0, -10), (0, 10), (0, 0)]
BROKEN = dict(scale_inputs=False, replay=False, target=False, double=False, lr=0.05)
FIXED = dict(scale_inputs=True, replay=True, target=True, double=True, lr=0.01)
def new_net(rng):
return {"w1": [[rng.gauss(0, 1.0) for _ in range(2)] for _ in range(H)], "b1": [0.0] * H,
"w2": [[rng.gauss(0, 0.3) for _ in range(H)] for _ in range(NA)], "b2": [0.0] * NA}
def copy_net(n):
return {"w1": [r[:] for r in n["w1"]], "b1": n["b1"][:], "w2": [r[:] for r in n["w2"]], "b2": n["b2"][:]}
def forward(n, x):
h = [math.tanh(n["w1"][j][0] * x[0] + n["w1"][j][1] * x[1] + n["b1"][j]) for j in range(H)]
return [sum(n["w2"][a][j] * h[j] for j in range(H)) + n["b2"][a] for a in range(NA)], h
def train_step(n, x, a, target, lr):
q, h = forward(n, x)
err = max(-1.0, min(1.0, q[a] - target)) # error clipping (kept in both configs)
for j in range(H):
g = err * n["w2"][a][j] * (1 - h[j] ** 2)
n["w2"][a][j] -= lr * err * h[j]
n["w1"][j][0] -= lr * g * x[0]
n["w1"][j][1] -= lr * g * x[1]
n["b1"][j] -= lr * g
n["b2"][a] -= lr * err
def step(s, a):
p = min(180.0, max(0.0, s[0] + MOVES[a][0]))
t = min(180.0, max(0.0, s[1] + MOVES[a][1]))
return (p, t), -math.dist((p, t), LAMP) / 50
def run(cfg, seed, episodes=120):
rng = random.Random(seed)
enc = (lambda s: (s[0] / 90 - 1, s[1] / 90 - 1)) if cfg["scale_inputs"] else (lambda s: s)
net = new_net(rng)
tgt, buf, updates = copy_net(net), [], 0
for ep in range(episodes):
s, eps = (rng.uniform(0, 180), rng.uniform(0, 180)), max(0.05, 1 - ep / (episodes * 0.6))
for _ in range(30):
q, _ = forward(net, enc(s))
a = rng.randrange(NA) if rng.random() < eps else max(range(NA), key=lambda i: q[i])
s2, r = step(s, a)
buf.append((enc(s), a, r, enc(s2)))
buf = buf[-4000:]
batch = rng.sample(buf, min(8, len(buf))) if cfg["replay"] else [buf[-1]]
for x, aa, rr, x2 in batch:
evaluator = tgt if cfg["target"] else net
if cfg["double"]:
qn, _ = forward(net, x2)
b = max(range(NA), key=lambda i: qn[i])
nxt = forward(evaluator, x2)[0][b]
else:
nxt = max(forward(evaluator, x2)[0])
train_step(net, x, aa, rr + GAMMA * nxt, cfg["lr"])
updates += 1
if cfg["target"] and updates % 500 == 0:
tgt = copy_net(net)
s = s2
ok, dists = 0, []
for _ in range(40):
s = (rng.uniform(0, 180), rng.uniform(0, 180))
for _ in range(30):
q, _ = forward(net, enc(s))
s, _ = step(s, max(range(NA), key=lambda i: q[i]))
d = math.dist(s, LAMP)
dists.append(d)
ok += d <= 15
return ok / 40, sum(dists) / 40
def report(name, cfg):
res = [run(cfg, s) for s in range(3)]
oks = " / ".join(f"{r[0] * 100:3.0f} %" for r in res)
print(f" {name:34s} {oks:24s} avg distance {sum(r[1] for r in res) / 3:5.1f} deg")
print("DQN challenge: pan-tilt light-seeker, lamp at (120, 60), 120 episodes, 3 seeds")
print(" config success (per seed)")
report("BROKEN (all four mistakes)", BROKEN)
cfg = dict(BROKEN)
for key, label in (("scale_inputs", "fix 1: scale inputs to -1..1"), ("lr", "fix 2: learning rate 0.05 -> 0.01"),
("replay", "fix 3: add experience replay"), ("target", "fix 4: add target network"),
("double", "bonus: Double DQN target")):
cfg[key] = FIXED[key]
report(label, cfg)Output (run with python3 dqn_challenge.py)
DQN challenge: pan-tilt light-seeker, lamp at (120, 60), 120 episodes, 3 seeds config success (per seed) BROKEN (all four mistakes) 0 % / 0 % / 0 % avg distance 85.7 deg fix 1: scale inputs to -1..1 15 % / 0 % / 72 % avg distance 33.8 deg fix 2: learning rate 0.05 -> 0.01 0 % / 0 % / 0 % avg distance 67.7 deg fix 3: add experience replay 50 % / 92 % / 0 % avg distance 22.2 deg fix 4: add target network 25 % / 100 % / 2 % avg distance 20.3 deg bonus: Double DQN target 35 % / 100 % / 5 % avg distance 17.5 deg
Try it: turn replay and the target sync on and off and train many batches. Which change calms the TD-error sparkline the most? (Educational simulation.)
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Bench build: a real two-servo light-seeker
pan_tilt_dqn.ino puts the whole recipe on an Arduino UNO: scaled inputs, a 2 → 8 → 5 network, a replay buffer of 32 with batches of 4, a target network, error clipping, ε-greedy exploration, and a Double DQN switch. The real robot uses the light-level reward from Lesson 2 (−0.1 per step, +1 brighter, −1 darker, +5 and terminated at level 3), on a 9 × 9 grid of 15° steps between 30° and 150°.
| Part | Connection | Note |
|---|---|---|
| Pan servo | signal → D9; red → separate 5–6 V; brown → supply GND | base servo |
| Tilt servo | signal → D10; red → separate 5–6 V; brown → supply GND | mounted on the pan horn |
| Servo supply | 5–6 V, at least 2 A for two micro servos | join supply GND to UNO GND |
| LDR | 5 V → LDR → A0 | mounted on the tilt bracket, facing forward |
| 10 kΩ resistor | A0 → GND | divider, as in Lesson 2 |
| Pause button | D2 → GND | uses INPUT_PULLUP |

// pan_tilt_dqn.ino - Stage 14 L10 challenge
// A mini DQN for a two-servo pan-tilt light-seeker. Everything from L06-L09 in one sketch:
// network 2 inputs (pan, tilt scaled to -1..1) -> 8 tanh -> 5 outputs (Q per action)
// replay buffer of 32, mini-batch 4, target network synced every SYNC updates,
// clipped error, optional Double DQN target (type d), epsilon-greedy.
// Actions: 0 pan-, 1 pan+, 2 tilt-, 3 tilt+, 4 stay (15 deg steps, 30..150 deg).
// Reward: -0.1 per step, +1 brighter, -1 darker, +5 and TERMINATED at light level 3.
// Serial commands: d = toggle Double DQN, s = slower sync (x2), f = faster sync (/2). Each resets.
//
// Wiring: pan servo signal D9, tilt servo signal D10, BOTH on their OWN 5-6 V supply
// (at least 2 A for two servos), supply GND joined to UNO GND. LDR 5V -> A0, 10k A0 -> GND,
// mounted on the tilt bracket. Button D2 -> GND = pause. Serial 115200.
// Never power servos from a GPIO pin or the UNO 5 V pin.
#include <Servo.h>
const int PAN_PIN = 9, TILT_PIN = 10, LDR_PIN = A0, BUTTON_PIN = 2;
const int H = 8, NA = 5, MAX_STEPS = 30, EPISODES = 80, BUF = 32, BATCH = 4;
const int LEVEL_EDGES[3] = {300, 550, 800}; // calibrate for your room (see L02)
const float LR = 0.02, GAMMA = 0.9;
const int8_t DPAN[NA] = {-1, 1, 0, 0, 0}, DTILT[NA] = {0, 0, -1, 1, 0};
struct Net { float w1[H][2], b1[H], w2[NA][H], b2[NA]; };
struct Transition { int8_t p, t, p2, t2; uint8_t a, done; float r; };
Net net, tgt;
Transition buf[BUF];
int bufCount = 0, bufNext = 0, episode = 0, syncEvery = 50, pan = 6, tilt = 4; // grid 0..8
unsigned long updates = 0;
bool paused = false, useDouble = false;
Servo panServo, tiltServo;
float sc(int g) { return (g - 4) / 4.0; } // grid 0..8 (30..150 deg) -> -1..1
int toDeg(int g) { return 30 + 15 * g; }
void forward(const Net &n, int p, int t, float q[NA], float h[H]) {
for (int j = 0; j < H; j++) h[j] = tanh(n.w1[j][0] * sc(p) + n.w1[j][1] * sc(t) + n.b1[j]);
for (int a = 0; a < NA; a++) { q[a] = n.b2[a]; for (int j = 0; j < H; j++) q[a] += n.w2[a][j] * h[j]; }
}
int argmax(const float q[NA]) { int b = 0; for (int a = 1; a < NA; a++) if (q[a] > q[b]) b = a; return b; }
void trainOn(const Transition &tr) {
float q[NA], h[H], qt[NA], ht[H];
float target = tr.r;
if (!tr.done) {
forward(tgt, tr.p2, tr.t2, qt, ht);
if (useDouble) { float qo[NA], ho[H]; forward(net, tr.p2, tr.t2, qo, ho); target += GAMMA * qt[argmax(qo)]; }
else target += GAMMA * qt[argmax(qt)];
}
forward(net, tr.p, tr.t, q, h);
float err = constrain(q[tr.a] - target, -1.0, 1.0);
for (int j = 0; j < H; j++) {
float g = err * net.w2[tr.a][j] * (1 - h[j] * h[j]);
net.w2[tr.a][j] -= LR * err * h[j];
net.w1[j][0] -= LR * g * sc(tr.p);
net.w1[j][1] -= LR * g * sc(tr.t);
net.b1[j] -= LR * g;
}
net.b2[tr.a] -= LR * err;
if (++updates % syncEvery == 0) tgt = net;
}
void resetAll() {
for (int j = 0; j < H; j++) {
net.w1[j][0] = random(-1000, 1001) / 500.0; net.w1[j][1] = random(-1000, 1001) / 500.0; net.b1[j] = 0;
for (int a = 0; a < NA; a++) net.w2[a][j] = random(-1000, 1001) / 3000.0;
}
for (int a = 0; a < NA; a++) net.b2[a] = 0;
tgt = net; bufCount = bufNext = 0; updates = 0; episode = 0;
Serial.print(F("# reset. double ")); Serial.print(useDouble ? F("on") : F("off"));
Serial.print(F(", sync every ")); Serial.println(syncEvery);
}
int readLevel() {
long s = 0;
for (int i = 0; i < 8; i++) { s += analogRead(LDR_PIN); delay(5); }
int raw = s / 8, level = 0;
while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
return level;
}
void moveServo(Servo &sv, int to) {
int from = sv.read(), dir = (to > from) ? 1 : -1;
for (int a = from; a != to; a += dir) { sv.write(a); delay(8); }
sv.write(to);
}
void moveTo(int p, int t) { moveServo(panServo, toDeg(p)); moveServo(tiltServo, toDeg(t)); delay(300); }
void runEpisode() {
float eps = max(0.05, 1.0 - episode / 40.0);
pan = random(9); tilt = random(9);
moveTo(pan, tilt);
int level = readLevel(), steps = 0;
float g = 0;
bool done = false;
while (steps < MAX_STEPS && !done) {
float q[NA], h[H];
forward(net, pan, tilt, q, h);
int a = (random(1000) < eps * 1000) ? random(NA) : argmax(q);
int p2 = constrain(pan + DPAN[a], 0, 8), t2 = constrain(tilt + DTILT[a], 0, 8);
if (p2 != pan || t2 != tilt) moveTo(p2, t2);
int level2 = readLevel();
float r = -0.1;
if (level2 > level) r = 1.0; else if (level2 < level) r = -1.0;
if (level2 == 3) { r = 5.0; done = true; }
Transition tr = {(int8_t)pan, (int8_t)tilt, (int8_t)p2, (int8_t)t2, (uint8_t)a, (uint8_t)done, r};
buf[bufNext] = tr; bufNext = (bufNext + 1) % BUF; if (bufCount < BUF) bufCount++;
for (int k = 0; k < BATCH; k++) trainOn(buf[random(bufCount)]);
pan = p2; tilt = t2; level = level2; g += r; steps++;
}
episode++;
Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
Serial.print(steps); Serial.print(','); Serial.print(g, 1); Serial.print(',');
Serial.println(done ? F("terminated") : F("truncated"));
}
void handleSerial() {
if (!Serial.available()) return;
char c = Serial.read();
if (c == 'd') useDouble = !useDouble;
else if (c == 's') syncEvery = min(syncEvery * 2, 3200);
else if (c == 'f') syncEvery = max(syncEvery / 2, 1);
else return;
resetAll();
}
void checkPause() {
if (digitalRead(BUTTON_PIN) == LOW) {
delay(30);
while (digitalRead(BUTTON_PIN) == LOW) {}
paused = !paused;
Serial.println(paused ? F("# paused") : F("# resumed"));
}
}
void setup() {
Serial.begin(115200);
pinMode(BUTTON_PIN, INPUT_PULLUP);
randomSeed(analogRead(A5));
panServo.attach(PAN_PIN); tiltServo.attach(TILT_PIN);
panServo.write(90); tiltServo.write(90);
delay(500);
Serial.println(F("episode,epsilon,steps,return,end"));
resetAll();
}
void loop() {
handleSerial();
checkPause();
if (paused) return;
if (episode < EPISODES) runEpisode();
}episode,epsilon,steps,return,end
# reset. double off, sync every 50
1,1.00,30,<return>,truncated
2,0.97,<steps>,<return>,terminated
...Debugging
| Symptom | Likely cause | Fix |
|---|---|---|
| UNO resets when both servos move | servos powered from the UNO | Separate 5–6 V supply, common ground |
| Never reaches level 3 | LEVEL_EDGES too high for your room, or lamp too far | Calibrate the edges (Lesson 2); bring the lamp closer |
| Head jitters at one spot | stay action learnt too early, or reading noise flips levels | Average more readings; add a small dead-band |
| Every episode truncated after 80 episodes | learning failed on this seed | Reset (type f then s) and run again; compare several runs |
| Python fix gives worse score | interaction with other settings (like fix 2) | Record it; try the fixes in a different order |
| Upload fails / random characters | wrong baud rate or board | Serial Monitor at 115200; select Arduino UNO |
Real world
Where this kind of debugging is real work

- DQN on Atari (2015): the DeepMind team combined a deep network with experience replay and a periodically updated target network; the paper describes both as ways to keep learning stable.
- Double DQN: adding the choose-with-one, value-with-the-other target to DQN outperforms the original DQN (Wikipedia, Q-learning).
- Ablation studies: in research, switching parts on and off one at a time — exactly our scoreboard — is how people show which ideas really matter.
- Your lab: a light-seeker today, a camera that tracks a ball or an arm that finds a part tomorrow. The debugging habits are the same.
Think
Think about it
Think
In the scoreboard, lowering the learning rate (fix 2) made things worse, but after replay was added, the low learning rate worked well. Why could the same change hurt in one setup and help in another?
Experiment
Run the challenge in the lab
About 90 minutes. Python 3 for the scoreboard; an Arduino UNO with two servos and an LDR for the bench test.
Experiment
Fix the broken DQN
You will need
- `dqn_challenge.py` (Python 3, no libraries)
- DQN intuition lab (in your browser)
- Arduino UNO, 2 micro servos on a pan-tilt bracket, separate 5–6 V ≥2 A supply
- LDR + 10 kΩ, push button, jumper wires
- Desk lamp on a stand
Steps
- Run `dqn_challenge.py` and copy the scoreboard into your notebook.
- Change the order: apply replay before the learning-rate fix. Does fix 2 still hurt?
- Change `range(3)` to `range(6)` in `report` for more seeds. Which numbers change most?
- Try sync every 100 and 2000 instead of 500 (search `updates % 500`). Record the scoreboard.
- Build the pan-tilt head, calibrate LEVEL_EDGES, and upload `pan_tilt_dqn.ino`. Log 80 episodes.
- Reset with Double DQN on (type `d`) and log 80 more. Compare average steps in the last 20 episodes.
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Each fix targets one problem: scaling helps the network’s units, replay reuses and de-correlates data, the target network stops the chasing, and Double DQN tames overestimation. Their effects depend on each other and on the seed, so a careful step-by-step log is the only way to know what really helped.
Knowledge check
Check your understanding
Knowledge check
1. Why does the challenge use a small network instead of a Q-table?
2. Scaling inputs from 0–180 to −1…1 helps because…
3. In the scoreboard, which single fix gave the biggest jump for two seeds?
4. Why did fix 2 (smaller learning rate) make things worse at first?
5. A target network…
6. Even with all fixes, seed 2 scored only 5 %. The right conclusion is…
7. Two servos should be powered from…
8. The most important debugging rule in this lesson is…
Mini assignment
Challenge report
Mini assignment
DQN challenge report
Fix the broken DQN, prove each step with numbers, and run the learner on a real pan-tilt head.
Deliverables
- Your scoreboard in the original order and in at least one different order, each with 6 seeds.
- A one-paragraph explanation for every fix: what problem it solves and what your numbers showed.
- Bench log: steps per episode for 80 episodes with plain targets and 80 with Double DQN, plotted together.
- A photo or sketch of your wiring showing the separate servo supply and common ground.
Do this, step by step
- Run
dqn_challenge.pyin the original order with 6 seeds (changerange(3)torange(6)). - Run it again in a different order of fixes and compare.
- Write one paragraph per fix: what it solves and what your numbers showed.
- Bench: pan and tilt servos on their own supply, grounds joined; calibrate
LEVEL_EDGESand uploadpan_tilt_dqn.ino. - Log 80 episodes with plain targets and 80 with Double DQN and plot them together.
Stuck? Hints, one at a time
- Change one key in
BROKENat a time; changing two at once hides which fix helped. - Input scaling matters first — raw angles of 30–150 make the tanh units saturate.
- Two servos can draw over 1 A when they stall — never power them from the UNO; use the separate supply.
Challenges (pick by level)
- 🟢 Beginner: Run the scoreboard and, for each line, write one sentence in your own words about what changed.
- 🟡 Intermediate: Increase episodes from 120 to 250. Does seed 2 recover with all fixes on?
- 🟠 Advanced: Make all 3 seeds reach at least 80 % success with the full config, changing only hyperparameters (hidden size, sync interval, ε schedule, batch size). Report exactly what you changed.
- 🔴 Expert: Add a soft target update (target ← 0.99·target + 0.01·online after every update) to the Python script and compare it with hard syncs every 500.
- ⚫ Research: Read the methods section of the DQN Nature paper and list which of its stability ideas appear in our script and which do not (for example, frame stacking and reward clipping).
Next step
You have now finished the DQN block of Stage 14: from why tables break to a working, honestly tested learner on real servos. Keep your challenge report — in the next part of the stage you will move from choosing among a few actions to learning smoother, continuous robot motions.
Look back: Lesson 9: Stability Traps & Overestimation. Next: Practice by stage. Related: Stage 12 — Simulation & Digital Twin.
Sources
- Mnih et al. (2015) — Human-level control through deep reinforcement learning (Nature) — DQN on Atari games.
- Mnih et al. (2015) — the same paper (PDF, DeepMind) — experience replay and a periodically updated target network for stability.
- Wikipedia — Q-learning (Deep Q-learning and Double Q-learning sections) — Double DQN outperforms the original DQN.
- OpenAI Baselines — DQN code (GitHub) — reference DQN implementation using the Huber loss (error clipping).
- [Sutton & Barto — Reinforcement Learning: An Introduction (2nd ed.)](http://incompleteideas.net/book/the-book-2nd.html) — Part II, approximate solution methods.
- Arduino Servo library (GitHub) — controlling RC hobby servos.
- Photos and diagrams: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).