Stage 14 · Lesson 08/10

Advanced Robotics / Reinforcement Learning · Lesson 08

DQN Core Ideas

Est. 70–85 min

Why

Why this lesson matters

A Deep Q-Network (DQN) is Q-learning where Q(s, a; w) is a neural network. In 2015, Mnih and colleagues at DeepMind showed that one DQN agent, receiving only the screen pixels and the game score, could reach a level comparable to a professional human games tester across 49 Atari 2600 games — with the same algorithm, network and settings for every game.

Plain Q-learning with a neural network is unstable. Today you will build the fix piece by piece in plain Python — experience replay and a target network — and measure what each piece adds on our servo task. Then you will run a mini DQN, with both tricks, on the Arduino light-seeker itself.

Concept

A network, a memory and a frozen copy

1. The network: Q-values for every action at once

The network takes the state as input (for us, one number: the servo angle; for the Atari DQN, screen pixels) and outputs one Q-value per action. Choosing the greedy action is just “pick the biggest output”. Learning is the same TD error as Lesson 4, used as the loss for gradient descent: make Q(s, a) closer to r + γ · max Q(s′, ·).

A diagram of a neural network with three input dots, a hidden layer of three circles and an output layer of two circles, all joined by arrows
A small multi-layer network. In a DQN the inputs are the state and there is one output per action, each giving its Q-value. Our servo network has 1 input, 16 hidden units and 3 outputs.Photo: Offnfopt · CC BY-SA 3.0 · Source: Wikimedia Commons · resized, converted to WebP

2. Why plain “neural Q-learning” is unstable

  • Correlated data: consecutive steps are almost the same state (89°, 90°, 91°…). Training on them in order is like revising only one chapter for a week — the network over-fits it and forgets the rest.
  • Small change, big effect: a small weight update can change the greedy policy, which changes which states the robot visits next, which changes the training data.
  • Moving target: the target r + γ · max Q(s′) is computed with the same network you are updating. Every step moves the goalposts.

Wikipedia’s Q-learning article lists exactly these three causes for why RL is unstable or divergent with a nonlinear function approximator such as a neural network.

3. Trick 1 — experience replay

Store every transition (s, a, r, s′, done) in a replay buffer. At each step, train on a random mini-batch drawn from the buffer instead of only the newest transition. Random samples break the correlation between neighbouring steps, and each experience is reused many times — valuable when every real robot step costs time and motor wear.

Hand-made English flash cards with pictures and words spread on a table
Shuffled flash cards: revising random old cards instead of only today’s lesson. A replay buffer does the same for the network.Photo: Tomchiukc (zh.wikipedia) · CC BY-SA 3.0 · Source: Wikimedia Commons · resized, converted to WebP

4. Trick 2 — the target network

Keep a frozen copy of the network. Compute targets r + γ · max Q_target(s′, ·) with the copy, and update only the online network. Every C updates, copy the online weights into the target. Now the goalposts stay still for a while, and the online network can actually reach them.

Several arrows clustered near the centre of a coloured archery target
Hitting a target that stays still is far easier than hitting one that moves every time you shoot. A target network holds the target still for C updates.Photo: Wacky1976 · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP
The DQN loop in eight lines (from dqn_core_lab.py) · Python
a = random_action() if rng.random() < eps else argmax(forward(net, s))
s2, r = env_step(s, a)
buf.append((s, a, r, s2))                                  # 1. remember
for s_, a_, r_, s2_ in rng.sample(buf, 16):               # 2. replay a random mini-batch
    target = r_ + GAMMA * max(forward(tgt, s2_))           # 3. target from the frozen copy
    train_step(net, s_, a_, target)                        # 4. gradient step on (Q - target)^2
    updates += 1
    if updates % 200 == 0: tgt = copy_net(net)             # 5. sync the target network

Visual

Visualise it: collect, replay, sync

The DQN loop as a toy: an online estimator, a replay buffer, mini-batch training and a lagged target that you sync by hand. Watch the TD-error sparkline as you train. (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Code lab

Code lab: build DQN one trick at a time

dqn_core_lab.py trains a 1 → 16 (tanh) → 3 network on the servo-hold task (move to 90° and stay; actions −5°, 0, +5°; 200 episodes × 40 steps; 3 seeds). Four versions: A online (newest transition only, targets from the same network), B + replay (mini-batch of 16 from a 5000-transition buffer), C + target network (synced every 200 updates), D both = DQN. Standard library only — about 10 seconds.

dqn_core_lab.py · Python
"""dqn_core_lab.py - Stage 14 L08: the core ideas of DQN, in plain Python.

A tiny neural network (1 input -> 16 hidden tanh -> 3 outputs, one Q per action) learns
the servo task from Lessons 6-7: move to 90 deg and HOLD (actions -5, 0, +5 deg).
No hand-made features: the raw angle (scaled to -1..1) goes straight in.

Four versions, same 200 episodes, 3 seeds:
  A) online: learn from each new transition once, target uses the same network
  B) + experience replay: store transitions, learn from random mini-batches of 16
  C) + target network: targets come from a frozen copy, synced every 200 updates
  D) DQN = replay + target network
Standard library only (slow but readable: about 10 seconds in total).
"""
import math
import random

H, NA, GAMMA, LR = 16, 3, 0.9, 0.01


def new_net(rng):
    return {"w1": [rng.gauss(0, 1.0) for _ in range(H)], "b1": [0.0] * H,
            "w2": [[rng.gauss(0, 0.3) for _ in range(H)] for _ in range(NA)], "b2": [0.0] * NA}


def copy_net(n):
    return {"w1": n["w1"][:], "b1": n["b1"][:], "w2": [r[:] for r in n["w2"]], "b2": n["b2"][:]}


def forward(n, x):
    h = [math.tanh(n["w1"][j] * x + n["b1"][j]) for j in range(H)]
    q = [sum(n["w2"][a][j] * h[j] for j in range(H)) + n["b2"][a] for a in range(NA)]
    return q, h


def train_step(n, x, a, target):
    """One gradient step on (Q(x,a) - target)^2 for the chosen action only."""
    q, h = forward(n, x)
    err = q[a] - target
    err = max(-1.0, min(1.0, err))                      # clip the error (Huber-like)
    for j in range(H):
        g_h = err * n["w2"][a][j] * (1 - h[j] ** 2)
        n["w2"][a][j] -= LR * err * h[j]
        n["w1"][j] -= LR * g_h * x
        n["b1"][j] -= LR * g_h
    n["b2"][a] -= LR * err


def scale(angle):
    return angle / 90 - 1                               # 0..180 deg -> -1..1


def run(replay, target_net, seed, episodes=200):
    rng = random.Random(seed)
    net = new_net(rng)
    tgt = copy_net(net)
    buf, updates = [], 0
    for ep in range(episodes):
        x, eps = rng.uniform(0, 180), max(0.05, 1 - ep / (episodes * 0.6))
        for _ in range(40):
            q, _ = forward(net, scale(x))
            a = rng.randrange(NA) if rng.random() < eps else max(range(NA), key=lambda i: q[i])
            x2 = min(180.0, max(0.0, x + (a - 1) * 5))
            r = -abs(x2 - 90) / 10
            buf.append((scale(x), a, r, scale(x2)))
            if len(buf) > 5000:
                buf.pop(0)
            batch = rng.sample(buf, min(16, len(buf))) if replay else [buf[-1]]
            for s, aa, rr, s2 in batch:
                q2, _ = forward(tgt if target_net else net, s2)
                train_step(net, s, aa, rr + GAMMA * max(q2))
                updates += 1
                if target_net and updates % 200 == 0:
                    tgt = copy_net(net)
            x = x2
    # greedy test
    errs = []
    for _ in range(50):
        x = rng.uniform(0, 180)
        for _ in range(40):
            q, _ = forward(net, scale(x))
            x = min(180.0, max(0.0, x + (max(range(NA), key=lambda i: q[i]) - 1) * 5))
        errs.append(abs(x - 90))
    qs = [forward(net, scale(v))[0] for v in (0, 45, 90, 135, 180)]
    return sum(1 for e in errs if e <= 5) / 50, sum(errs) / 50, qs, updates


print("DQN core ideas on the servo-hold task (200 episodes x 40 steps, 3 seeds)")
print("   version                        updates   ends within 5 deg (per seed)   avg error")
for name, rep, tn in (("A) online, same network", False, False), ("B) + experience replay", True, False),
                      ("C) + target network", False, True), ("D) DQN: replay + target", True, True)):
    res = [run(rep, tn, s) for s in range(3)]
    oks = " / ".join(f"{r[0] * 100:3.0f} %" for r in res)
    err = sum(r[1] for r in res) / 3
    print(f"   {name:28s} {res[0][3]:8d}      {oks:24s}      {err:5.1f} deg")
    if name.startswith("D"):
        best = res[0]
print("\nWhat the DQN (seed 0) learnt - greedy action at a few angles:")
for v, q in zip((0, 45, 90, 135, 180), best[2]):
    a = max(range(NA), key=lambda i: q[i])
    print(f"   {v:3d} deg   Q = [{q[0]:6.2f} {q[1]:6.2f} {q[2]:6.2f}]   -> {['-5 deg', 'stay', '+5 deg'][a]}")

Output (run with python3 dqn_core_lab.py)

DQN core ideas on the servo-hold task (200 episodes x 40 steps, 3 seeds)
   version                        updates   ends within 5 deg (per seed)   avg error
   A) online, same network          8000        0 % /   0 % /  48 %          16.7 deg
   B) + experience replay         127880       48 % /  92 % /   0 %          32.5 deg
   C) + target network              8000       72 % /  84 % /   0 %          11.8 deg
   D) DQN: replay + target        127880       94 % /  98 % /  58 %           2.9 deg

What the DQN (seed 0) learnt - greedy action at a few angles:
     0 deg   Q = [-50.97 -49.21 -45.42]   -> +5 deg
    45 deg   Q = [-22.22 -18.87 -16.16]   -> +5 deg
    90 deg   Q = [ -1.86  -1.83  -2.04]   -> stay
   135 deg   Q = [-15.14 -18.02 -21.54]   -> -5 deg
   180 deg   Q = [-45.31 -48.73 -51.48]   -> -5 deg

Try it: collect transitions into the replay buffer, train on mini-batches, and press Sync to copy the online weights to the target. Try training many batches without syncing — what happens to the TD error? (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Bench build: a mini DQN on the UNO

mini_dqn_light_seeker.ino runs the full recipe on the light-seeker: a 1 → 8 (tanh) → 3 network, a 32-transition replay buffer with mini-batches of 4, and a target network copied every 50 updates. Type r or t in the Serial Monitor to switch replay or the target network off (or back on) — the network resets so you can compare.

Safety: comparing versions means several 60-episode runs — use a separate servo supply (never a GPIO pin or the UNO 5 V pin), keep fingers away from the horn, and pause (D2) to let a warm servo rest.
PartConnectionNote
SG90/MG90-class servosignal → D9the same light-seeker head
Servo supplyred → separate 5–6 V, brown → supply GNDjoin supply GND to UNO GND
LDR + 10 kΩLDR 5 V → A0, 10 kΩ A0 → GNDcalibrate LEVEL_EDGES (Lesson 2)
Push buttonD2 → GND (internal pull-up)pause / resume
mini_dqn_light_seeker.ino · Arduino C++
// mini_dqn_light_seeker.ino - Stage 14 L08
// The DQN recipe, shrunk to fit an Arduino UNO, on the light-seeker from L04/L07:
//   * a tiny neural network  1 input (angle) -> 8 hidden (tanh) -> 3 outputs (Q per action)
//   * experience replay      the last 32 transitions are stored; each step trains on 4 random ones
//   * target network         a frozen copy gives the targets; copied every 50 updates
// State = servo angle 30..150 deg (10 deg steps). Actions: 0 LEFT, 1 STAY, 2 RIGHT.
// Reward as L04: -0.1 per step, +1 brighter, -1 darker, +5 and TERMINATED at light level 3.
// Type r in the Serial Monitor to toggle replay, t to toggle the target network (then reset).
//
// Wiring: same as L01-L07. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = pause / resume. Serial 115200.
// Never power the servo from a GPIO pin or the UNO 5 V pin.
#include <Servo.h>

const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int H = 8, NA = 3, MAX_STEPS = 25, EPISODES = 60, BUF = 32, BATCH = 4, SYNC = 50;
const int LEVEL_EDGES[3] = {300, 550, 800};       // calibrate for your room (see L02)
const float LR = 0.02, GAMMA = 0.9;

struct Net { float w1[H], b1[H], w2[NA][H], b2[NA]; };
struct Transition { int8_t x; int8_t x2; uint8_t a; uint8_t done; float r; };

Net net, tgt;                                      // 2 x 43 floats = 344 bytes
Transition buf[BUF];                               // 32 x 8 bytes = 256 bytes
int bufCount = 0, bufNext = 0, angle = 90, episode = 0;
unsigned long updates = 0;
bool paused = false, useReplay = true, useTarget = true;
Servo head;

float scaleAngle(int deg) { return (deg - 90) / 60.0; }      // 30..150 -> -1..1

void forward(const Net &n, float x, float q[NA], float h[H]) {
  for (int j = 0; j < H; j++) h[j] = tanh(n.w1[j] * x + n.b1[j]);
  for (int a = 0; a < NA; a++) {
    q[a] = n.b2[a];
    for (int j = 0; j < H; j++) q[a] += n.w2[a][j] * h[j];
  }
}

float maxOf(const float q[NA]) { return max(q[0], max(q[1], q[2])); }

void trainOn(const Transition &t) {
  float q[NA], h[H], q2[NA], h2[H];
  forward(useTarget ? tgt : net, scaleAngle(t.x2 * 10), q2, h2);
  float target = t.done ? t.r : t.r + GAMMA * maxOf(q2);
  forward(net, scaleAngle(t.x * 10), q, h);
  float err = constrain(q[t.a] - target, -1.0, 1.0);          // clipped error
  float x = scaleAngle(t.x * 10);
  for (int j = 0; j < H; j++) {
    float gh = err * net.w2[t.a][j] * (1 - h[j] * h[j]);
    net.w2[t.a][j] -= LR * err * h[j];
    net.w1[j] -= LR * gh * x;
    net.b1[j] -= LR * gh;
  }
  net.b2[t.a] -= LR * err;
  updates++;
  if (useTarget && updates % SYNC == 0) tgt = net;             // sync the target network
}

void resetNet() {
  for (int j = 0; j < H; j++) {
    net.w1[j] = random(-1000, 1001) / 500.0; net.b1[j] = 0;
    for (int a = 0; a < NA; a++) net.w2[a][j] = random(-1000, 1001) / 3000.0;
  }
  for (int a = 0; a < NA; a++) net.b2[a] = 0;
  tgt = net; bufCount = 0; bufNext = 0; updates = 0; episode = 0;
}

int readLevel() {
  long sum = 0;
  for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
  int raw = sum / 8, level = 0;
  while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
  return level;
}

void moveTo(int to) {
  int from = head.read(), dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
  head.write(to);
  delay(300);
}

int greedy(int deg) {
  float q[NA], h[H];
  forward(net, scaleAngle(deg), q, h);
  int b = 0;
  for (int a = 1; a < NA; a++) if (q[a] > q[b]) b = a;
  return b;
}

void runEpisode() {
  float eps = max(0.05, 1.0 - episode / 30.0);
  angle = 30 + 10 * random(13);
  moveTo(angle);
  int level = readLevel(), t = 0;
  float g = 0;
  bool done = false;
  while (t < MAX_STEPS && !done) {
    int a = (random(1000) < eps * 1000) ? random(NA) : greedy(angle);
    int a2 = constrain(angle + (a - 1) * 10, 30, 150);
    if (a2 != angle) moveTo(a2);
    int level2 = readLevel();
    float r = -0.1;
    if (level2 > level) r = 1.0; else if (level2 < level) r = -1.0;
    if (level2 == 3) { r = 5.0; done = true; }
    Transition tr = {(int8_t)(angle / 10), (int8_t)(a2 / 10), (uint8_t)a, (uint8_t)done, r};
    buf[bufNext] = tr; bufNext = (bufNext + 1) % BUF; if (bufCount < BUF) bufCount++;
    if (useReplay) { for (int k = 0; k < BATCH; k++) trainOn(buf[random(bufCount)]); }
    else trainOn(tr);                                         // learn only from the newest step
    angle = a2; level = level2; g += r; t++;
  }
  episode++;
  Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
  Serial.print(t); Serial.print(','); Serial.print(g, 1); Serial.print(',');
  Serial.print(updates); Serial.print(','); Serial.println(done ? F("terminated") : F("truncated"));
}

void printPolicy() {
  Serial.print(F("# greedy policy (replay ")); Serial.print(useReplay ? F("on") : F("off"));
  Serial.print(F(", target ")); Serial.print(useTarget ? F("on") : F("off")); Serial.println(F("):"));
  for (int deg = 30; deg <= 150; deg += 10) {
    int b = greedy(deg);
    Serial.print(deg); Serial.println(b == 0 ? F(" -> LEFT") : (b == 1 ? F(" -> STAY") : F(" -> RIGHT")));
  }
}

void handleSerial() {
  if (!Serial.available()) return;
  char c = Serial.read();
  if (c == 'r') useReplay = !useReplay;
  else if (c == 't') useTarget = !useTarget;
  else return;
  resetNet();
  Serial.print(F("# reset. replay ")); Serial.print(useReplay ? F("on") : F("off"));
  Serial.print(F(", target ")); Serial.println(useTarget ? F("on") : F("off"));
}

void checkPause() {
  if (digitalRead(BUTTON_PIN) == LOW) {
    delay(30);
    while (digitalRead(BUTTON_PIN) == LOW) {}
    paused = !paused;
    Serial.println(paused ? F("# paused") : F("# resumed"));
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  randomSeed(analogRead(A5));
  head.attach(SERVO_PIN);
  head.write(90);
  delay(500);
  resetNet();
  Serial.println(F("episode,epsilon,steps,return,updates,end"));
}

void loop() {
  handleSerial();
  checkPause();
  if (paused) return;
  if (episode < EPISODES) {
    runEpisode();
    if (episode % 10 == 0) printPolicy();
  }
}
Serial Monitor (115200) — shape of the output (your numbers will differ) · Text
episode,epsilon,steps,return,updates,end
1,1.00,<steps>,<return>,<updates>,<terminated/truncated>
...
# greedy policy (replay on, target on):
30 -> RIGHT
...
150 -> LEFT
# reset. replay off, target on

Debugging

SymptomLikely causeFix
Q-values explode to huge numbers or nanlearning rate too high, no error clippingLower LR; clip the error to ±1
Learns, then suddenly collapsesno target network, or syncing too oftenAdd the target net; sync less often (bigger C)
Never learns anythingbuffer too small or inputs not scaledScale inputs to about −1…+1; collect more before training
Greedy policy same everywherehidden units saturated (huge inputs or weights)Scale inputs; smaller initial weights
Results change a lot between seedsnormal for RLReport several seeds; compare averages
UNO resets when the servo movesservo on UNO 5 VSeparate supply, grounds joined

Real world

DQN in the real world

A black Atari 2600 video game console with four switches on the front
An Atari 2600, first released in 1977. DQN learnt to play 49 games made for this console from pixels and score alone (Mnih et al., Nature, 2015).Photo: Evan-Amos · Public domain · Source: Wikimedia Commons · resized, converted to WebP
  • Games from pixels: the Nature DQN used a deep convolutional network on screen images, and the same settings across 49 games.
  • Robots: the Nature paper’s references include Long-Ji Lin’s 1993 report “Reinforcement learning for robots using neural networks” — replaying past experience to learn efficiently is an old robotics idea.
  • Discrete actions only: DQN picks from a list of actions (left/stay/right). Robot arms with continuous torques usually need other algorithms built on the same ideas (replay buffers and target networks appear again there).
  • Practice: on real robots, replay is precious because every real transition costs time and wear; many teams train in simulation (Stage 12) and fine-tune on hardware.

Think

Think about it

Think

Your replay buffer holds only the last 32 transitions, and the robot spent the last 32 steps standing at 90°. What does a random mini-batch from this buffer teach the network? What could go wrong?

Experiment

Try it in the lab

About 70 minutes. Python 3 for the four versions; the light-seeker hardware for the mini DQN.

Experiment

Remove one trick at a time

You will need

  • `dqn_core_lab.py` (Python 3, no libraries)
  • DQN intuition lab (in your browser)
  • Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
  • LED torch on a stand

Steps

  1. Run `dqn_core_lab.py`. Copy the table and the learnt Q-values into your notebook.
  2. Change the target sync from every 200 updates to every 10, then every 2000. Run version D each time. Which is best?
  3. Change the mini-batch from 16 to 1 in version D. Is it still better than version C?
  4. Make version A fairer: train on the newest transition 16 times per step. Does it catch up with D?
  5. On the bench, run 60 episodes with both tricks on. Then type r (replay off) and run again; then r and t (replay on, target off).
  6. In the DQN intuition lab, train many batches without pressing Sync, then Sync and train again. Compare the TD-error sparkline.

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

A neural network alone generalises but is unstable. Replay makes the data look more like a shuffled, varied dataset and reuses experience. A target network keeps the goalposts still. Together they turn a fragile learner into one that works on most seeds.

Knowledge check

Check your understanding

Knowledge check

1. In a DQN, the neural network outputs…

2. Experience replay means…

3. The target network is…

4. Which is NOT a cause of instability listed on Wikipedia for Q-learning with neural networks?

5. In our Python lab, which version did best on every seed?

6. The 2015 Nature DQN learnt Atari games from…

7. Why scale the servo angle to −1…+1 before the network?

8. Why is replay especially valuable on a real robot?

Mini assignment

Hands-on assignment

Mini assignment

DQN ablation report

Prove, with your own runs, what each DQN ingredient does.

Deliverables

  • A results table for versions A–D with at least 5 seeds each (change `range(3)` to `range(5)`), with average and worst seed.
  • A graph of target-sync interval (10, 50, 200, 1000, 2000) vs average “within 5°” for version D.
  • The bench comparison: first episode with reliable “terminated” and the final greedy policy for (replay on, target on), (off, on) and (on, off).
  • A half-page explanation, in your own words, of why replay and the target network help — use the flash-card and archery analogies or your own.

Do this, step by step

  1. In dqn_core_lab.py change range(3) to range(5) and run versions A–D.
  2. Fill the table with average and worst seed for each version.
  3. For version D, try target-sync intervals 10, 50, 200, 1000 and 2000 and graph them.
  4. Bench: upload mini_dqn_light_seeker.ino; type r or t to switch replay or the target network.
  5. Write your half-page explanation of replay and the target network.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Report the worst seed as well as the average — one lucky seed can hide an unstable method.
  2. The sketch resets the network whenever you type r or t, so each comparison starts fresh.
  3. Replay BUF is only 32 on the UNO — small, but enough to break the order of consecutive steps.

Challenges (pick by level)

  • 🟢 Beginner: Draw the DQN loop as a flow chart: act → store → sample → target → update → sync. Label where ε, γ and C appear.
  • 🟡 Intermediate: Add a “warm-up”: do not train until the buffer has 500 transitions. Does version D improve or get worse?
  • 🟠 Advanced: Give the network two inputs — angle and the last action — and see whether it learns faster.
  • 🔴 Expert: Replace the periodic copy with a soft update w⁻ ← 0.99 w⁻ + 0.01 w after every update. Compare with C = 200.
  • ⚫ Research: Read the abstract of Mnih et al. (2015) and its Extended Data Table 3 caption (“effects of replay and separating the target Q-network”). Compare their conclusion with your table.

Next step

DQN works — but it has traps. The max in the target picks the luckiest noisy estimate, so Q-values tend to be too high; and a few settings can still make training blow up. In Lesson 9 — Stability Traps & Overestimation you will measure overestimation and fix it with Double Q-learning.

Look back: Lesson 7: Function Approximation Intuition. Next: Lesson 9: Stability Traps & Overestimation. Related: Stage 12 — Simulation & Digital Twin.

Sources

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in