Advanced Robotics / Reinforcement Learning · Lesson 08
DQN Core Ideas
Est. 70–85 min
Why
Why this lesson matters
A Deep Q-Network (DQN) is Q-learning where Q(s, a; w) is a neural network. In 2015, Mnih and colleagues at DeepMind showed that one DQN agent, receiving only the screen pixels and the game score, could reach a level comparable to a professional human games tester across 49 Atari 2600 games — with the same algorithm, network and settings for every game.
Plain Q-learning with a neural network is unstable. Today you will build the fix piece by piece in plain Python — experience replay and a target network — and measure what each piece adds on our servo task. Then you will run a mini DQN, with both tricks, on the Arduino light-seeker itself.
Concept
A network, a memory and a frozen copy
1. The network: Q-values for every action at once
The network takes the state as input (for us, one number: the servo angle; for the Atari DQN, screen pixels) and outputs one Q-value per action. Choosing the greedy action is just “pick the biggest output”. Learning is the same TD error as Lesson 4, used as the loss for gradient descent: make Q(s, a) closer to r + γ · max Q(s′, ·).

2. Why plain “neural Q-learning” is unstable
- Correlated data: consecutive steps are almost the same state (89°, 90°, 91°…). Training on them in order is like revising only one chapter for a week — the network over-fits it and forgets the rest.
- Small change, big effect: a small weight update can change the greedy policy, which changes which states the robot visits next, which changes the training data.
- Moving target: the target r + γ · max Q(s′) is computed with the same network you are updating. Every step moves the goalposts.
Wikipedia’s Q-learning article lists exactly these three causes for why RL is unstable or divergent with a nonlinear function approximator such as a neural network.
3. Trick 1 — experience replay
Store every transition (s, a, r, s′, done) in a replay buffer. At each step, train on a random mini-batch drawn from the buffer instead of only the newest transition. Random samples break the correlation between neighbouring steps, and each experience is reused many times — valuable when every real robot step costs time and motor wear.

4. Trick 2 — the target network
Keep a frozen copy of the network. Compute targets r + γ · max Q_target(s′, ·) with the copy, and update only the online network. Every C updates, copy the online weights into the target. Now the goalposts stay still for a while, and the online network can actually reach them.

a = random_action() if rng.random() < eps else argmax(forward(net, s))
s2, r = env_step(s, a)
buf.append((s, a, r, s2)) # 1. remember
for s_, a_, r_, s2_ in rng.sample(buf, 16): # 2. replay a random mini-batch
target = r_ + GAMMA * max(forward(tgt, s2_)) # 3. target from the frozen copy
train_step(net, s_, a_, target) # 4. gradient step on (Q - target)^2
updates += 1
if updates % 200 == 0: tgt = copy_net(net) # 5. sync the target networkVisual
Visualise it: collect, replay, sync
The DQN loop as a toy: an online estimator, a replay buffer, mini-batch training and a lagged target that you sync by hand. Watch the TD-error sparkline as you train. (Educational simulation.)
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Code lab
Code lab: build DQN one trick at a time
dqn_core_lab.py trains a 1 → 16 (tanh) → 3 network on the servo-hold task (move to 90° and stay; actions −5°, 0, +5°; 200 episodes × 40 steps; 3 seeds). Four versions: A online (newest transition only, targets from the same network), B + replay (mini-batch of 16 from a 5000-transition buffer), C + target network (synced every 200 updates), D both = DQN. Standard library only — about 10 seconds.
"""dqn_core_lab.py - Stage 14 L08: the core ideas of DQN, in plain Python.
A tiny neural network (1 input -> 16 hidden tanh -> 3 outputs, one Q per action) learns
the servo task from Lessons 6-7: move to 90 deg and HOLD (actions -5, 0, +5 deg).
No hand-made features: the raw angle (scaled to -1..1) goes straight in.
Four versions, same 200 episodes, 3 seeds:
A) online: learn from each new transition once, target uses the same network
B) + experience replay: store transitions, learn from random mini-batches of 16
C) + target network: targets come from a frozen copy, synced every 200 updates
D) DQN = replay + target network
Standard library only (slow but readable: about 10 seconds in total).
"""
import math
import random
H, NA, GAMMA, LR = 16, 3, 0.9, 0.01
def new_net(rng):
return {"w1": [rng.gauss(0, 1.0) for _ in range(H)], "b1": [0.0] * H,
"w2": [[rng.gauss(0, 0.3) for _ in range(H)] for _ in range(NA)], "b2": [0.0] * NA}
def copy_net(n):
return {"w1": n["w1"][:], "b1": n["b1"][:], "w2": [r[:] for r in n["w2"]], "b2": n["b2"][:]}
def forward(n, x):
h = [math.tanh(n["w1"][j] * x + n["b1"][j]) for j in range(H)]
q = [sum(n["w2"][a][j] * h[j] for j in range(H)) + n["b2"][a] for a in range(NA)]
return q, h
def train_step(n, x, a, target):
"""One gradient step on (Q(x,a) - target)^2 for the chosen action only."""
q, h = forward(n, x)
err = q[a] - target
err = max(-1.0, min(1.0, err)) # clip the error (Huber-like)
for j in range(H):
g_h = err * n["w2"][a][j] * (1 - h[j] ** 2)
n["w2"][a][j] -= LR * err * h[j]
n["w1"][j] -= LR * g_h * x
n["b1"][j] -= LR * g_h
n["b2"][a] -= LR * err
def scale(angle):
return angle / 90 - 1 # 0..180 deg -> -1..1
def run(replay, target_net, seed, episodes=200):
rng = random.Random(seed)
net = new_net(rng)
tgt = copy_net(net)
buf, updates = [], 0
for ep in range(episodes):
x, eps = rng.uniform(0, 180), max(0.05, 1 - ep / (episodes * 0.6))
for _ in range(40):
q, _ = forward(net, scale(x))
a = rng.randrange(NA) if rng.random() < eps else max(range(NA), key=lambda i: q[i])
x2 = min(180.0, max(0.0, x + (a - 1) * 5))
r = -abs(x2 - 90) / 10
buf.append((scale(x), a, r, scale(x2)))
if len(buf) > 5000:
buf.pop(0)
batch = rng.sample(buf, min(16, len(buf))) if replay else [buf[-1]]
for s, aa, rr, s2 in batch:
q2, _ = forward(tgt if target_net else net, s2)
train_step(net, s, aa, rr + GAMMA * max(q2))
updates += 1
if target_net and updates % 200 == 0:
tgt = copy_net(net)
x = x2
# greedy test
errs = []
for _ in range(50):
x = rng.uniform(0, 180)
for _ in range(40):
q, _ = forward(net, scale(x))
x = min(180.0, max(0.0, x + (max(range(NA), key=lambda i: q[i]) - 1) * 5))
errs.append(abs(x - 90))
qs = [forward(net, scale(v))[0] for v in (0, 45, 90, 135, 180)]
return sum(1 for e in errs if e <= 5) / 50, sum(errs) / 50, qs, updates
print("DQN core ideas on the servo-hold task (200 episodes x 40 steps, 3 seeds)")
print(" version updates ends within 5 deg (per seed) avg error")
for name, rep, tn in (("A) online, same network", False, False), ("B) + experience replay", True, False),
("C) + target network", False, True), ("D) DQN: replay + target", True, True)):
res = [run(rep, tn, s) for s in range(3)]
oks = " / ".join(f"{r[0] * 100:3.0f} %" for r in res)
err = sum(r[1] for r in res) / 3
print(f" {name:28s} {res[0][3]:8d} {oks:24s} {err:5.1f} deg")
if name.startswith("D"):
best = res[0]
print("\nWhat the DQN (seed 0) learnt - greedy action at a few angles:")
for v, q in zip((0, 45, 90, 135, 180), best[2]):
a = max(range(NA), key=lambda i: q[i])
print(f" {v:3d} deg Q = [{q[0]:6.2f} {q[1]:6.2f} {q[2]:6.2f}] -> {['-5 deg', 'stay', '+5 deg'][a]}")Output (run with python3 dqn_core_lab.py)
DQN core ideas on the servo-hold task (200 episodes x 40 steps, 3 seeds)
version updates ends within 5 deg (per seed) avg error
A) online, same network 8000 0 % / 0 % / 48 % 16.7 deg
B) + experience replay 127880 48 % / 92 % / 0 % 32.5 deg
C) + target network 8000 72 % / 84 % / 0 % 11.8 deg
D) DQN: replay + target 127880 94 % / 98 % / 58 % 2.9 deg
What the DQN (seed 0) learnt - greedy action at a few angles:
0 deg Q = [-50.97 -49.21 -45.42] -> +5 deg
45 deg Q = [-22.22 -18.87 -16.16] -> +5 deg
90 deg Q = [ -1.86 -1.83 -2.04] -> stay
135 deg Q = [-15.14 -18.02 -21.54] -> -5 deg
180 deg Q = [-45.31 -48.73 -51.48] -> -5 degTry it: collect transitions into the replay buffer, train on mini-batches, and press Sync to copy the online weights to the target. Try training many batches without syncing — what happens to the TD error? (Educational simulation.)
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Bench build: a mini DQN on the UNO
mini_dqn_light_seeker.ino runs the full recipe on the light-seeker: a 1 → 8 (tanh) → 3 network, a 32-transition replay buffer with mini-batches of 4, and a target network copied every 50 updates. Type r or t in the Serial Monitor to switch replay or the target network off (or back on) — the network resets so you can compare.
| Part | Connection | Note |
|---|---|---|
| SG90/MG90-class servo | signal → D9 | the same light-seeker head |
| Servo supply | red → separate 5–6 V, brown → supply GND | join supply GND to UNO GND |
| LDR + 10 kΩ | LDR 5 V → A0, 10 kΩ A0 → GND | calibrate LEVEL_EDGES (Lesson 2) |
| Push button | D2 → GND (internal pull-up) | pause / resume |
// mini_dqn_light_seeker.ino - Stage 14 L08
// The DQN recipe, shrunk to fit an Arduino UNO, on the light-seeker from L04/L07:
// * a tiny neural network 1 input (angle) -> 8 hidden (tanh) -> 3 outputs (Q per action)
// * experience replay the last 32 transitions are stored; each step trains on 4 random ones
// * target network a frozen copy gives the targets; copied every 50 updates
// State = servo angle 30..150 deg (10 deg steps). Actions: 0 LEFT, 1 STAY, 2 RIGHT.
// Reward as L04: -0.1 per step, +1 brighter, -1 darker, +5 and TERMINATED at light level 3.
// Type r in the Serial Monitor to toggle replay, t to toggle the target network (then reset).
//
// Wiring: same as L01-L07. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = pause / resume. Serial 115200.
// Never power the servo from a GPIO pin or the UNO 5 V pin.
#include <Servo.h>
const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int H = 8, NA = 3, MAX_STEPS = 25, EPISODES = 60, BUF = 32, BATCH = 4, SYNC = 50;
const int LEVEL_EDGES[3] = {300, 550, 800}; // calibrate for your room (see L02)
const float LR = 0.02, GAMMA = 0.9;
struct Net { float w1[H], b1[H], w2[NA][H], b2[NA]; };
struct Transition { int8_t x; int8_t x2; uint8_t a; uint8_t done; float r; };
Net net, tgt; // 2 x 43 floats = 344 bytes
Transition buf[BUF]; // 32 x 8 bytes = 256 bytes
int bufCount = 0, bufNext = 0, angle = 90, episode = 0;
unsigned long updates = 0;
bool paused = false, useReplay = true, useTarget = true;
Servo head;
float scaleAngle(int deg) { return (deg - 90) / 60.0; } // 30..150 -> -1..1
void forward(const Net &n, float x, float q[NA], float h[H]) {
for (int j = 0; j < H; j++) h[j] = tanh(n.w1[j] * x + n.b1[j]);
for (int a = 0; a < NA; a++) {
q[a] = n.b2[a];
for (int j = 0; j < H; j++) q[a] += n.w2[a][j] * h[j];
}
}
float maxOf(const float q[NA]) { return max(q[0], max(q[1], q[2])); }
void trainOn(const Transition &t) {
float q[NA], h[H], q2[NA], h2[H];
forward(useTarget ? tgt : net, scaleAngle(t.x2 * 10), q2, h2);
float target = t.done ? t.r : t.r + GAMMA * maxOf(q2);
forward(net, scaleAngle(t.x * 10), q, h);
float err = constrain(q[t.a] - target, -1.0, 1.0); // clipped error
float x = scaleAngle(t.x * 10);
for (int j = 0; j < H; j++) {
float gh = err * net.w2[t.a][j] * (1 - h[j] * h[j]);
net.w2[t.a][j] -= LR * err * h[j];
net.w1[j] -= LR * gh * x;
net.b1[j] -= LR * gh;
}
net.b2[t.a] -= LR * err;
updates++;
if (useTarget && updates % SYNC == 0) tgt = net; // sync the target network
}
void resetNet() {
for (int j = 0; j < H; j++) {
net.w1[j] = random(-1000, 1001) / 500.0; net.b1[j] = 0;
for (int a = 0; a < NA; a++) net.w2[a][j] = random(-1000, 1001) / 3000.0;
}
for (int a = 0; a < NA; a++) net.b2[a] = 0;
tgt = net; bufCount = 0; bufNext = 0; updates = 0; episode = 0;
}
int readLevel() {
long sum = 0;
for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
int raw = sum / 8, level = 0;
while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
return level;
}
void moveTo(int to) {
int from = head.read(), dir = (to > from) ? 1 : -1;
for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
head.write(to);
delay(300);
}
int greedy(int deg) {
float q[NA], h[H];
forward(net, scaleAngle(deg), q, h);
int b = 0;
for (int a = 1; a < NA; a++) if (q[a] > q[b]) b = a;
return b;
}
void runEpisode() {
float eps = max(0.05, 1.0 - episode / 30.0);
angle = 30 + 10 * random(13);
moveTo(angle);
int level = readLevel(), t = 0;
float g = 0;
bool done = false;
while (t < MAX_STEPS && !done) {
int a = (random(1000) < eps * 1000) ? random(NA) : greedy(angle);
int a2 = constrain(angle + (a - 1) * 10, 30, 150);
if (a2 != angle) moveTo(a2);
int level2 = readLevel();
float r = -0.1;
if (level2 > level) r = 1.0; else if (level2 < level) r = -1.0;
if (level2 == 3) { r = 5.0; done = true; }
Transition tr = {(int8_t)(angle / 10), (int8_t)(a2 / 10), (uint8_t)a, (uint8_t)done, r};
buf[bufNext] = tr; bufNext = (bufNext + 1) % BUF; if (bufCount < BUF) bufCount++;
if (useReplay) { for (int k = 0; k < BATCH; k++) trainOn(buf[random(bufCount)]); }
else trainOn(tr); // learn only from the newest step
angle = a2; level = level2; g += r; t++;
}
episode++;
Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
Serial.print(t); Serial.print(','); Serial.print(g, 1); Serial.print(',');
Serial.print(updates); Serial.print(','); Serial.println(done ? F("terminated") : F("truncated"));
}
void printPolicy() {
Serial.print(F("# greedy policy (replay ")); Serial.print(useReplay ? F("on") : F("off"));
Serial.print(F(", target ")); Serial.print(useTarget ? F("on") : F("off")); Serial.println(F("):"));
for (int deg = 30; deg <= 150; deg += 10) {
int b = greedy(deg);
Serial.print(deg); Serial.println(b == 0 ? F(" -> LEFT") : (b == 1 ? F(" -> STAY") : F(" -> RIGHT")));
}
}
void handleSerial() {
if (!Serial.available()) return;
char c = Serial.read();
if (c == 'r') useReplay = !useReplay;
else if (c == 't') useTarget = !useTarget;
else return;
resetNet();
Serial.print(F("# reset. replay ")); Serial.print(useReplay ? F("on") : F("off"));
Serial.print(F(", target ")); Serial.println(useTarget ? F("on") : F("off"));
}
void checkPause() {
if (digitalRead(BUTTON_PIN) == LOW) {
delay(30);
while (digitalRead(BUTTON_PIN) == LOW) {}
paused = !paused;
Serial.println(paused ? F("# paused") : F("# resumed"));
}
}
void setup() {
Serial.begin(115200);
pinMode(BUTTON_PIN, INPUT_PULLUP);
randomSeed(analogRead(A5));
head.attach(SERVO_PIN);
head.write(90);
delay(500);
resetNet();
Serial.println(F("episode,epsilon,steps,return,updates,end"));
}
void loop() {
handleSerial();
checkPause();
if (paused) return;
if (episode < EPISODES) {
runEpisode();
if (episode % 10 == 0) printPolicy();
}
}episode,epsilon,steps,return,updates,end
1,1.00,<steps>,<return>,<updates>,<terminated/truncated>
...
# greedy policy (replay on, target on):
30 -> RIGHT
...
150 -> LEFT
# reset. replay off, target onDebugging
| Symptom | Likely cause | Fix |
|---|---|---|
| Q-values explode to huge numbers or nan | learning rate too high, no error clipping | Lower LR; clip the error to ±1 |
| Learns, then suddenly collapses | no target network, or syncing too often | Add the target net; sync less often (bigger C) |
| Never learns anything | buffer too small or inputs not scaled | Scale inputs to about −1…+1; collect more before training |
| Greedy policy same everywhere | hidden units saturated (huge inputs or weights) | Scale inputs; smaller initial weights |
| Results change a lot between seeds | normal for RL | Report several seeds; compare averages |
| UNO resets when the servo moves | servo on UNO 5 V | Separate supply, grounds joined |
Real world
DQN in the real world

- Games from pixels: the Nature DQN used a deep convolutional network on screen images, and the same settings across 49 games.
- Robots: the Nature paper’s references include Long-Ji Lin’s 1993 report “Reinforcement learning for robots using neural networks” — replaying past experience to learn efficiently is an old robotics idea.
- Discrete actions only: DQN picks from a list of actions (left/stay/right). Robot arms with continuous torques usually need other algorithms built on the same ideas (replay buffers and target networks appear again there).
- Practice: on real robots, replay is precious because every real transition costs time and wear; many teams train in simulation (Stage 12) and fine-tune on hardware.
Think
Think about it
Think
Your replay buffer holds only the last 32 transitions, and the robot spent the last 32 steps standing at 90°. What does a random mini-batch from this buffer teach the network? What could go wrong?
Experiment
Try it in the lab
About 70 minutes. Python 3 for the four versions; the light-seeker hardware for the mini DQN.
Experiment
Remove one trick at a time
You will need
- `dqn_core_lab.py` (Python 3, no libraries)
- DQN intuition lab (in your browser)
- Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
- LED torch on a stand
Steps
- Run `dqn_core_lab.py`. Copy the table and the learnt Q-values into your notebook.
- Change the target sync from every 200 updates to every 10, then every 2000. Run version D each time. Which is best?
- Change the mini-batch from 16 to 1 in version D. Is it still better than version C?
- Make version A fairer: train on the newest transition 16 times per step. Does it catch up with D?
- On the bench, run 60 episodes with both tricks on. Then type r (replay off) and run again; then r and t (replay on, target off).
- In the DQN intuition lab, train many batches without pressing Sync, then Sync and train again. Compare the TD-error sparkline.
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
A neural network alone generalises but is unstable. Replay makes the data look more like a shuffled, varied dataset and reuses experience. A target network keeps the goalposts still. Together they turn a fragile learner into one that works on most seeds.
Knowledge check
Check your understanding
Knowledge check
1. In a DQN, the neural network outputs…
2. Experience replay means…
3. The target network is…
4. Which is NOT a cause of instability listed on Wikipedia for Q-learning with neural networks?
5. In our Python lab, which version did best on every seed?
6. The 2015 Nature DQN learnt Atari games from…
7. Why scale the servo angle to −1…+1 before the network?
8. Why is replay especially valuable on a real robot?
Mini assignment
Hands-on assignment
Mini assignment
DQN ablation report
Prove, with your own runs, what each DQN ingredient does.
Deliverables
- A results table for versions A–D with at least 5 seeds each (change `range(3)` to `range(5)`), with average and worst seed.
- A graph of target-sync interval (10, 50, 200, 1000, 2000) vs average “within 5°” for version D.
- The bench comparison: first episode with reliable “terminated” and the final greedy policy for (replay on, target on), (off, on) and (on, off).
- A half-page explanation, in your own words, of why replay and the target network help — use the flash-card and archery analogies or your own.
Do this, step by step
- In
dqn_core_lab.pychangerange(3)torange(5)and run versions A–D. - Fill the table with average and worst seed for each version.
- For version D, try target-sync intervals 10, 50, 200, 1000 and 2000 and graph them.
- Bench: upload
mini_dqn_light_seeker.ino; type r or t to switch replay or the target network. - Write your half-page explanation of replay and the target network.
Stuck? Hints, one at a time
- Report the worst seed as well as the average — one lucky seed can hide an unstable method.
- The sketch resets the network whenever you type r or t, so each comparison starts fresh.
- Replay
BUFis only 32 on the UNO — small, but enough to break the order of consecutive steps.
Challenges (pick by level)
- 🟢 Beginner: Draw the DQN loop as a flow chart: act → store → sample → target → update → sync. Label where ε, γ and C appear.
- 🟡 Intermediate: Add a “warm-up”: do not train until the buffer has 500 transitions. Does version D improve or get worse?
- 🟠 Advanced: Give the network two inputs — angle and the last action — and see whether it learns faster.
- 🔴 Expert: Replace the periodic copy with a soft update w⁻ ← 0.99 w⁻ + 0.01 w after every update. Compare with C = 200.
- ⚫ Research: Read the abstract of Mnih et al. (2015) and its Extended Data Table 3 caption (“effects of replay and separating the target Q-network”). Compare their conclusion with your table.
Next step
DQN works — but it has traps. The max in the target picks the luckiest noisy estimate, so Q-values tend to be too high; and a few settings can still make training blow up. In Lesson 9 — Stability Traps & Overestimation you will measure overestimation and fix it with Double Q-learning.
Look back: Lesson 7: Function Approximation Intuition. Next: Lesson 9: Stability Traps & Overestimation. Related: Stage 12 — Simulation & Digital Twin.
Sources
- Mnih et al. (2015) — Human-level control through deep reinforcement learning, Nature 518 — deep Q-network from pixels and game score; 49 Atari 2600 games with the same algorithm, architecture and hyperparameters; level comparable to a professional human games tester; Breakout tunnelling video after 600 episodes; Extended Data Table 3 on replay and the separate target network; reference to Lin (1993) robots and neural networks.
- Mnih et al. (2015) — full paper with Methods (PDF, DeepMind) — experience replay: each step used in many updates, random samples break correlations, last N tuples sampled uniformly; separate target network for stability.
- OpenAI Baselines — DQN code (GitHub) — reference DQN implementation; uses the Huber loss, the usual way to apply the Nature paper’s error clipping to −1…1.
- Wikipedia — Q-learning (Deep Q-learning section) — three causes of instability with nonlinear approximators; experience replay as random samples of prior actions; targets updated only periodically.
- Wikipedia — Atari 2600 — the console (released c. September 1977).
- [Sutton & Barto — Reinforcement Learning: An Introduction (2nd ed., free online)](http://incompleteideas.net/book/the-book-2nd.html) — Part II approximate methods; Chapter 16 case studies include Atari game play.
- Arduino Servo library (GitHub) — controlling RC hobby servos.
- Photos and diagrams: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).