Stage 14 · Lesson 10/10

Advanced Robotics / Reinforcement Learning · Lesson 10

DQN Intuition Challenge

Est. 75–95 min

Why

Why this challenge matters

Lessons 6–9 gave you the pieces one by one: why a Q-table explodes, how a function can generalise, how experience replay and a target network make DQN stable, and why max() overestimates. This lesson is a challenge. You get one broken DQN with four classic mistakes, a scoreboard, and a real two-servo robot. Your job is to fix it step by step, measure every step honestly over several seeds, and explain *why* each fix helped — or did not.

This is exactly how robot-learning engineers work. Nobody writes a perfect learner in one go. They change one thing at a time, keep a log, and trust numbers from several seeds rather than one lucky run.

Concept

The challenge brief: fix a broken DQN

1. The robot: a pan-tilt light-seeker

Two hobby servos make a “head”: one turns left-right (pan), one tilts up-down. The state is just two angles, (pan, tilt), each 0–180°. In the simulation the lamp sits at (120°, 60°). There are 5 actions: pan −10°, pan +10°, tilt −10°, tilt +10°, stay. Each step the reward is −(distance to lamp) ÷ 50, for 30 steps. Success = ending within 15° of the lamp.

A white dome security camera mounted on a ceiling tile
A pan-tilt-zoom camera: two motors aim one sensor, exactly like our two-servo light-seeker. Its state is “where am I pointing”, and its job is to point at something interesting.Photo: Wikideas1 · CC0 · Source: Wikimedia Commons · resized, converted to WebP

Why not a table? With 10° bins you need 19 × 19 positions × 5 actions = 1805 values, each learnt separately, and the robot still cannot tell 61° from 69°. A tiny network (2 inputs → 24 tanh → 5 Q-values) shares what it learns between nearby angles — the Lesson 7 idea.

2. The four classic mistakes in BROKEN

The original DQN paper used experience replay and a separate, periodically updated target network to make learning with a neural network stable (Mnih et al., 2015).
MistakeWhy it hurtsFix (lesson)
Raw inputs 0–180tanh units saturate; gradients are tiny or hugescale to −1…1 (L7)
Learning rate 0.05big jumps; values swingsmaller step, 0.01 (L9)
Learn only from the last stepconsecutive steps are almost the same, so the network over-fits the latest momentexperience replay, mini-batch of 8 (L8)
Target from the same networkthe goalpost moves on every updatetarget network synced every 500 updates (L8)
(bonus) plain max targetmax of noisy Q-values is too highDouble DQN target (L9)
The two config switches (from dqn_challenge.py) · Python
BROKEN = dict(scale_inputs=False, replay=False, target=False, double=False, lr=0.05)
FIXED = dict(scale_inputs=True, replay=True, target=True, double=True, lr=0.01)

3. The rules of the challenge

  1. One change at a time. If you change two things and it gets better, you do not know which one helped.
  2. Several seeds. A tiny DQN is fragile. One seed can lie — in both directions.
  3. Test greedily. Training return includes exploration. Score the learnt policy with ε = 0 from fresh random starts (the script uses 40).
  4. Write down surprises. A fix that makes things worse is data, not failure.
A black desk lamp with a flexible neck on a table
The “goal” of our challenge: an ordinary desk lamp. Put it on a stand and move it between runs so the robot cannot just memorise one spot by luck.Photo: AirbusA330772673 · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

Visual

Visualise it: the whole DQN loop in one place

Everything from Lessons 6–9 in one picture: a small Q-network, a replay buffer and a lagged target network you can sync by hand. (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Code lab

Code lab: the scoreboard

dqn_challenge.py trains the BROKEN config, then switches the fixes on cumulatively in this order: scale inputs → lower learning rate → replay (buffer 4000, batch 8) → target network (sync every 500 updates) → Double DQN target. Each config runs 120 episodes on 3 seeds and is scored greedily from 40 random starts. Standard library only; it took about 9 seconds on our machine.

dqn_challenge.py · Python
"""dqn_challenge.py - Stage 14 L10: the DQN intuition challenge.

Task: a pan-tilt light-seeker. State = (pan, tilt) in degrees, 0..180 each.
The lamp is at (120, 60). Actions: pan -10/+10, tilt -10/+10, stay (5 actions).
Reward = -distance_to_lamp / 50 each step; 30 steps per episode, no early stop.
Success = ending within 15 deg of the lamp.

A Q-table with 10 deg bins would need 19 x 19 x 5 = 1805 values, each learnt separately.
Instead we use a tiny DQN (2 inputs -> 24 tanh -> 5). The BROKEN config below has
four classic mistakes. Fix them one at a time and watch the scoreboard.
Standard library only (about 10 seconds).
"""
import math
import random

H, NA, GAMMA = 24, 5, 0.9
LAMP = (120.0, 60.0)
MOVES = [(-10, 0), (10, 0), (0, -10), (0, 10), (0, 0)]

BROKEN = dict(scale_inputs=False, replay=False, target=False, double=False, lr=0.05)
FIXED = dict(scale_inputs=True, replay=True, target=True, double=True, lr=0.01)


def new_net(rng):
    return {"w1": [[rng.gauss(0, 1.0) for _ in range(2)] for _ in range(H)], "b1": [0.0] * H,
            "w2": [[rng.gauss(0, 0.3) for _ in range(H)] for _ in range(NA)], "b2": [0.0] * NA}


def copy_net(n):
    return {"w1": [r[:] for r in n["w1"]], "b1": n["b1"][:], "w2": [r[:] for r in n["w2"]], "b2": n["b2"][:]}


def forward(n, x):
    h = [math.tanh(n["w1"][j][0] * x[0] + n["w1"][j][1] * x[1] + n["b1"][j]) for j in range(H)]
    return [sum(n["w2"][a][j] * h[j] for j in range(H)) + n["b2"][a] for a in range(NA)], h


def train_step(n, x, a, target, lr):
    q, h = forward(n, x)
    err = max(-1.0, min(1.0, q[a] - target))           # error clipping (kept in both configs)
    for j in range(H):
        g = err * n["w2"][a][j] * (1 - h[j] ** 2)
        n["w2"][a][j] -= lr * err * h[j]
        n["w1"][j][0] -= lr * g * x[0]
        n["w1"][j][1] -= lr * g * x[1]
        n["b1"][j] -= lr * g
    n["b2"][a] -= lr * err


def step(s, a):
    p = min(180.0, max(0.0, s[0] + MOVES[a][0]))
    t = min(180.0, max(0.0, s[1] + MOVES[a][1]))
    return (p, t), -math.dist((p, t), LAMP) / 50


def run(cfg, seed, episodes=120):
    rng = random.Random(seed)
    enc = (lambda s: (s[0] / 90 - 1, s[1] / 90 - 1)) if cfg["scale_inputs"] else (lambda s: s)
    net = new_net(rng)
    tgt, buf, updates = copy_net(net), [], 0
    for ep in range(episodes):
        s, eps = (rng.uniform(0, 180), rng.uniform(0, 180)), max(0.05, 1 - ep / (episodes * 0.6))
        for _ in range(30):
            q, _ = forward(net, enc(s))
            a = rng.randrange(NA) if rng.random() < eps else max(range(NA), key=lambda i: q[i])
            s2, r = step(s, a)
            buf.append((enc(s), a, r, enc(s2)))
            buf = buf[-4000:]
            batch = rng.sample(buf, min(8, len(buf))) if cfg["replay"] else [buf[-1]]
            for x, aa, rr, x2 in batch:
                evaluator = tgt if cfg["target"] else net
                if cfg["double"]:
                    qn, _ = forward(net, x2)
                    b = max(range(NA), key=lambda i: qn[i])
                    nxt = forward(evaluator, x2)[0][b]
                else:
                    nxt = max(forward(evaluator, x2)[0])
                train_step(net, x, aa, rr + GAMMA * nxt, cfg["lr"])
                updates += 1
                if cfg["target"] and updates % 500 == 0:
                    tgt = copy_net(net)
            s = s2
    ok, dists = 0, []
    for _ in range(40):
        s = (rng.uniform(0, 180), rng.uniform(0, 180))
        for _ in range(30):
            q, _ = forward(net, enc(s))
            s, _ = step(s, max(range(NA), key=lambda i: q[i]))
        d = math.dist(s, LAMP)
        dists.append(d)
        ok += d <= 15
    return ok / 40, sum(dists) / 40


def report(name, cfg):
    res = [run(cfg, s) for s in range(3)]
    oks = " / ".join(f"{r[0] * 100:3.0f} %" for r in res)
    print(f"   {name:34s} {oks:24s}  avg distance {sum(r[1] for r in res) / 3:5.1f} deg")


print("DQN challenge: pan-tilt light-seeker, lamp at (120, 60), 120 episodes, 3 seeds")
print("   config                             success (per seed)")
report("BROKEN (all four mistakes)", BROKEN)
cfg = dict(BROKEN)
for key, label in (("scale_inputs", "fix 1: scale inputs to -1..1"), ("lr", "fix 2: learning rate 0.05 -> 0.01"),
                   ("replay", "fix 3: add experience replay"), ("target", "fix 4: add target network"),
                   ("double", "bonus: Double DQN target")):
    cfg[key] = FIXED[key]
    report(label, cfg)

Output (run with python3 dqn_challenge.py)

DQN challenge: pan-tilt light-seeker, lamp at (120, 60), 120 episodes, 3 seeds
   config                             success (per seed)
   BROKEN (all four mistakes)           0 % /   0 % /   0 %     avg distance  85.7 deg
   fix 1: scale inputs to -1..1        15 % /   0 % /  72 %     avg distance  33.8 deg
   fix 2: learning rate 0.05 -> 0.01    0 % /   0 % /   0 %     avg distance  67.7 deg
   fix 3: add experience replay        50 % /  92 % /   0 %     avg distance  22.2 deg
   fix 4: add target network           25 % / 100 % /   2 %     avg distance  20.3 deg
   bonus: Double DQN target            35 % / 100 % /   5 %     avg distance  17.5 deg

Try it: turn replay and the target sync on and off and train many batches. Which change calms the TD-error sparkline the most? (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Bench build: a real two-servo light-seeker

pan_tilt_dqn.ino puts the whole recipe on an Arduino UNO: scaled inputs, a 2 → 8 → 5 network, a replay buffer of 32 with batches of 4, a target network, error clipping, ε-greedy exploration, and a Double DQN switch. The real robot uses the light-level reward from Lesson 2 (−0.1 per step, +1 brighter, −1 darker, +5 and terminated at level 3), on a 9 × 9 grid of 15° steps between 30° and 150°.

Safety: never power servos from a GPIO pin or the UNO’s 5 V pin — two servos starting together can draw far more current than the board can give, causing resets or damage. Use a separate supply with a common ground. Keep fingers clear of the moving head and the lamp’s hot parts; keep any mains lamp wiring well away from the breadboard.
PartConnectionNote
Pan servosignal → D9; red → separate 5–6 V; brown → supply GNDbase servo
Tilt servosignal → D10; red → separate 5–6 V; brown → supply GNDmounted on the pan horn
Servo supply5–6 V, at least 2 A for two micro servosjoin supply GND to UNO GND
LDR5 V → LDR → A0mounted on the tilt bracket, facing forward
10 kΩ resistorA0 → GNDdivider, as in Lesson 2
Pause buttonD2 → GNDuses INPUT_PULLUP
A small blue micro servo with a white horn and an orange, red and brown cable
A micro servo of the SG90 type (Commons describes this particular unit as a counterfeit Tower Pro). Two of these on a pan-tilt bracket make our light-seeker head. Cheap servos vary — check yours turns smoothly before training.Photo: Suyash Dwivedi · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP
pan_tilt_dqn.ino · Arduino C++
// pan_tilt_dqn.ino - Stage 14 L10 challenge
// A mini DQN for a two-servo pan-tilt light-seeker. Everything from L06-L09 in one sketch:
//   network 2 inputs (pan, tilt scaled to -1..1) -> 8 tanh -> 5 outputs (Q per action)
//   replay buffer of 32, mini-batch 4, target network synced every SYNC updates,
//   clipped error, optional Double DQN target (type d), epsilon-greedy.
// Actions: 0 pan-, 1 pan+, 2 tilt-, 3 tilt+, 4 stay (15 deg steps, 30..150 deg).
// Reward: -0.1 per step, +1 brighter, -1 darker, +5 and TERMINATED at light level 3.
// Serial commands: d = toggle Double DQN, s = slower sync (x2), f = faster sync (/2). Each resets.
//
// Wiring: pan servo signal D9, tilt servo signal D10, BOTH on their OWN 5-6 V supply
// (at least 2 A for two servos), supply GND joined to UNO GND. LDR 5V -> A0, 10k A0 -> GND,
// mounted on the tilt bracket. Button D2 -> GND = pause. Serial 115200.
// Never power servos from a GPIO pin or the UNO 5 V pin.
#include <Servo.h>

const int PAN_PIN = 9, TILT_PIN = 10, LDR_PIN = A0, BUTTON_PIN = 2;
const int H = 8, NA = 5, MAX_STEPS = 30, EPISODES = 80, BUF = 32, BATCH = 4;
const int LEVEL_EDGES[3] = {300, 550, 800};       // calibrate for your room (see L02)
const float LR = 0.02, GAMMA = 0.9;
const int8_t DPAN[NA] = {-1, 1, 0, 0, 0}, DTILT[NA] = {0, 0, -1, 1, 0};

struct Net { float w1[H][2], b1[H], w2[NA][H], b2[NA]; };
struct Transition { int8_t p, t, p2, t2; uint8_t a, done; float r; };

Net net, tgt;
Transition buf[BUF];
int bufCount = 0, bufNext = 0, episode = 0, syncEvery = 50, pan = 6, tilt = 4;   // grid 0..8
unsigned long updates = 0;
bool paused = false, useDouble = false;
Servo panServo, tiltServo;

float sc(int g) { return (g - 4) / 4.0; }          // grid 0..8 (30..150 deg) -> -1..1
int toDeg(int g) { return 30 + 15 * g; }

void forward(const Net &n, int p, int t, float q[NA], float h[H]) {
  for (int j = 0; j < H; j++) h[j] = tanh(n.w1[j][0] * sc(p) + n.w1[j][1] * sc(t) + n.b1[j]);
  for (int a = 0; a < NA; a++) { q[a] = n.b2[a]; for (int j = 0; j < H; j++) q[a] += n.w2[a][j] * h[j]; }
}

int argmax(const float q[NA]) { int b = 0; for (int a = 1; a < NA; a++) if (q[a] > q[b]) b = a; return b; }

void trainOn(const Transition &tr) {
  float q[NA], h[H], qt[NA], ht[H];
  float target = tr.r;
  if (!tr.done) {
    forward(tgt, tr.p2, tr.t2, qt, ht);
    if (useDouble) { float qo[NA], ho[H]; forward(net, tr.p2, tr.t2, qo, ho); target += GAMMA * qt[argmax(qo)]; }
    else target += GAMMA * qt[argmax(qt)];
  }
  forward(net, tr.p, tr.t, q, h);
  float err = constrain(q[tr.a] - target, -1.0, 1.0);
  for (int j = 0; j < H; j++) {
    float g = err * net.w2[tr.a][j] * (1 - h[j] * h[j]);
    net.w2[tr.a][j] -= LR * err * h[j];
    net.w1[j][0] -= LR * g * sc(tr.p);
    net.w1[j][1] -= LR * g * sc(tr.t);
    net.b1[j] -= LR * g;
  }
  net.b2[tr.a] -= LR * err;
  if (++updates % syncEvery == 0) tgt = net;
}

void resetAll() {
  for (int j = 0; j < H; j++) {
    net.w1[j][0] = random(-1000, 1001) / 500.0; net.w1[j][1] = random(-1000, 1001) / 500.0; net.b1[j] = 0;
    for (int a = 0; a < NA; a++) net.w2[a][j] = random(-1000, 1001) / 3000.0;
  }
  for (int a = 0; a < NA; a++) net.b2[a] = 0;
  tgt = net; bufCount = bufNext = 0; updates = 0; episode = 0;
  Serial.print(F("# reset. double ")); Serial.print(useDouble ? F("on") : F("off"));
  Serial.print(F(", sync every ")); Serial.println(syncEvery);
}

int readLevel() {
  long s = 0;
  for (int i = 0; i < 8; i++) { s += analogRead(LDR_PIN); delay(5); }
  int raw = s / 8, level = 0;
  while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
  return level;
}

void moveServo(Servo &sv, int to) {
  int from = sv.read(), dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { sv.write(a); delay(8); }
  sv.write(to);
}

void moveTo(int p, int t) { moveServo(panServo, toDeg(p)); moveServo(tiltServo, toDeg(t)); delay(300); }

void runEpisode() {
  float eps = max(0.05, 1.0 - episode / 40.0);
  pan = random(9); tilt = random(9);
  moveTo(pan, tilt);
  int level = readLevel(), steps = 0;
  float g = 0;
  bool done = false;
  while (steps < MAX_STEPS && !done) {
    float q[NA], h[H];
    forward(net, pan, tilt, q, h);
    int a = (random(1000) < eps * 1000) ? random(NA) : argmax(q);
    int p2 = constrain(pan + DPAN[a], 0, 8), t2 = constrain(tilt + DTILT[a], 0, 8);
    if (p2 != pan || t2 != tilt) moveTo(p2, t2);
    int level2 = readLevel();
    float r = -0.1;
    if (level2 > level) r = 1.0; else if (level2 < level) r = -1.0;
    if (level2 == 3) { r = 5.0; done = true; }
    Transition tr = {(int8_t)pan, (int8_t)tilt, (int8_t)p2, (int8_t)t2, (uint8_t)a, (uint8_t)done, r};
    buf[bufNext] = tr; bufNext = (bufNext + 1) % BUF; if (bufCount < BUF) bufCount++;
    for (int k = 0; k < BATCH; k++) trainOn(buf[random(bufCount)]);
    pan = p2; tilt = t2; level = level2; g += r; steps++;
  }
  episode++;
  Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
  Serial.print(steps); Serial.print(','); Serial.print(g, 1); Serial.print(',');
  Serial.println(done ? F("terminated") : F("truncated"));
}

void handleSerial() {
  if (!Serial.available()) return;
  char c = Serial.read();
  if (c == 'd') useDouble = !useDouble;
  else if (c == 's') syncEvery = min(syncEvery * 2, 3200);
  else if (c == 'f') syncEvery = max(syncEvery / 2, 1);
  else return;
  resetAll();
}

void checkPause() {
  if (digitalRead(BUTTON_PIN) == LOW) {
    delay(30);
    while (digitalRead(BUTTON_PIN) == LOW) {}
    paused = !paused;
    Serial.println(paused ? F("# paused") : F("# resumed"));
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  randomSeed(analogRead(A5));
  panServo.attach(PAN_PIN); tiltServo.attach(TILT_PIN);
  panServo.write(90); tiltServo.write(90);
  delay(500);
  Serial.println(F("episode,epsilon,steps,return,end"));
  resetAll();
}

void loop() {
  handleSerial();
  checkPause();
  if (paused) return;
  if (episode < EPISODES) runEpisode();
}
Serial Monitor (115200) — shape of the output (your numbers will differ) · Text
episode,epsilon,steps,return,end
# reset. double off, sync every 50
1,1.00,30,<return>,truncated
2,0.97,<steps>,<return>,terminated
...

Debugging

SymptomLikely causeFix
UNO resets when both servos moveservos powered from the UNOSeparate 5–6 V supply, common ground
Never reaches level 3LEVEL_EDGES too high for your room, or lamp too farCalibrate the edges (Lesson 2); bring the lamp closer
Head jitters at one spotstay action learnt too early, or reading noise flips levelsAverage more readings; add a small dead-band
Every episode truncated after 80 episodeslearning failed on this seedReset (type f then s) and run again; compare several runs
Python fix gives worse scoreinteraction with other settings (like fix 2)Record it; try the fixes in a different order
Upload fails / random characterswrong baud rate or boardSerial Monitor at 115200; select Arduino UNO

Real world

Where this kind of debugging is real work

Rows of orange industrial robot arms working on car bodies in a factory
Industrial robots in a car body shop. Before any learnt behaviour goes near machines like these, engineers test it many times in simulation, one change at a time — the same discipline as this challenge.Photo: BMW Werk Leipzig · CC BY-SA 2.0 de · Source: Wikimedia Commons · resized, converted to WebP
  • DQN on Atari (2015): the DeepMind team combined a deep network with experience replay and a periodically updated target network; the paper describes both as ways to keep learning stable.
  • Double DQN: adding the choose-with-one, value-with-the-other target to DQN outperforms the original DQN (Wikipedia, Q-learning).
  • Ablation studies: in research, switching parts on and off one at a time — exactly our scoreboard — is how people show which ideas really matter.
  • Your lab: a light-seeker today, a camera that tracks a ball or an arm that finds a part tomorrow. The debugging habits are the same.

Think

Think about it

Think

In the scoreboard, lowering the learning rate (fix 2) made things worse, but after replay was added, the low learning rate worked well. Why could the same change hurt in one setup and help in another?

Experiment

Run the challenge in the lab

About 90 minutes. Python 3 for the scoreboard; an Arduino UNO with two servos and an LDR for the bench test.

Experiment

Fix the broken DQN

You will need

  • `dqn_challenge.py` (Python 3, no libraries)
  • DQN intuition lab (in your browser)
  • Arduino UNO, 2 micro servos on a pan-tilt bracket, separate 5–6 V ≥2 A supply
  • LDR + 10 kΩ, push button, jumper wires
  • Desk lamp on a stand

Steps

  1. Run `dqn_challenge.py` and copy the scoreboard into your notebook.
  2. Change the order: apply replay before the learning-rate fix. Does fix 2 still hurt?
  3. Change `range(3)` to `range(6)` in `report` for more seeds. Which numbers change most?
  4. Try sync every 100 and 2000 instead of 500 (search `updates % 500`). Record the scoreboard.
  5. Build the pan-tilt head, calibrate LEVEL_EDGES, and upload `pan_tilt_dqn.ino`. Log 80 episodes.
  6. Reset with Double DQN on (type `d`) and log 80 more. Compare average steps in the last 20 episodes.

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Each fix targets one problem: scaling helps the network’s units, replay reuses and de-correlates data, the target network stops the chasing, and Double DQN tames overestimation. Their effects depend on each other and on the seed, so a careful step-by-step log is the only way to know what really helped.

Knowledge check

Check your understanding

Knowledge check

1. Why does the challenge use a small network instead of a Q-table?

2. Scaling inputs from 0–180 to −1…1 helps because…

3. In the scoreboard, which single fix gave the biggest jump for two seeds?

4. Why did fix 2 (smaller learning rate) make things worse at first?

5. A target network…

6. Even with all fixes, seed 2 scored only 5 %. The right conclusion is…

7. Two servos should be powered from…

8. The most important debugging rule in this lesson is…

Mini assignment

Challenge report

Mini assignment

DQN challenge report

Fix the broken DQN, prove each step with numbers, and run the learner on a real pan-tilt head.

Deliverables

  • Your scoreboard in the original order and in at least one different order, each with 6 seeds.
  • A one-paragraph explanation for every fix: what problem it solves and what your numbers showed.
  • Bench log: steps per episode for 80 episodes with plain targets and 80 with Double DQN, plotted together.
  • A photo or sketch of your wiring showing the separate servo supply and common ground.

Do this, step by step

  1. Run dqn_challenge.py in the original order with 6 seeds (change range(3) to range(6)).
  2. Run it again in a different order of fixes and compare.
  3. Write one paragraph per fix: what it solves and what your numbers showed.
  4. Bench: pan and tilt servos on their own supply, grounds joined; calibrate LEVEL_EDGES and upload pan_tilt_dqn.ino.
  5. Log 80 episodes with plain targets and 80 with Double DQN and plot them together.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Change one key in BROKEN at a time; changing two at once hides which fix helped.
  2. Input scaling matters first — raw angles of 30–150 make the tanh units saturate.
  3. Two servos can draw over 1 A when they stall — never power them from the UNO; use the separate supply.

Challenges (pick by level)

  • 🟢 Beginner: Run the scoreboard and, for each line, write one sentence in your own words about what changed.
  • 🟡 Intermediate: Increase episodes from 120 to 250. Does seed 2 recover with all fixes on?
  • 🟠 Advanced: Make all 3 seeds reach at least 80 % success with the full config, changing only hyperparameters (hidden size, sync interval, ε schedule, batch size). Report exactly what you changed.
  • 🔴 Expert: Add a soft target update (target ← 0.99·target + 0.01·online after every update) to the Python script and compare it with hard syncs every 500.
  • ⚫ Research: Read the methods section of the DQN Nature paper and list which of its stability ideas appear in our script and which do not (for example, frame stacking and reward clipping).

Next step

You have now finished the DQN block of Stage 14: from why tables break to a working, honestly tested learner on real servos. Keep your challenge report — in the next part of the stage you will move from choosing among a few actions to learning smoother, continuous robot motions.

Look back: Lesson 9: Stability Traps & Overestimation. Next: Practice by stage. Related: Stage 12 — Simulation & Digital Twin.

Sources

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in