Advanced Robotics / Reinforcement Learning · Lesson 09
Stability Traps & Overestimation
Est. 65–80 min
Why
Why this lesson matters
Q-learning’s target uses max over Q(s′, ·). When the Q-values are noisy estimates, the max tends to pick an action whose estimate happened to come out too high. So targets are biased upward — overestimation. Wikipedia’s Q-learning article puts it this way: because the same Q function both selects and evaluates the next action, Q-learning can overestimate action values in noisy environments, slowing learning. Double Q-learning fixes this by selecting with one estimate and evaluating with another.
Today you will measure the max bias with numbers, watch Q-learning get fooled on a gripper decision while Double Q-learning is not, probe the classic DQN stability traps (learning rate, clipping, target sync), and see real overestimation from your own LDR’s noise on the bench.
Concept
Why the max lies, and how training goes wrong
1. The max of noisy guesses is too high
Suppose all actions are truly worth 0, but each estimate has random noise. The max of the estimates is almost always above 0 — and the more actions, the bigger the error. In Part 1 below (noise with standard deviation 1): 2 actions → +0.57, 10 actions → +1.55, 50 actions → +2.24, even though the true best value is 0.

2. Double Q-learning: choose with one, value with the other
Keep two estimates, Q_A and Q_B, each trained on different experiences (flip a coin each step). To update Q_A, let Q_A choose the best next action, but let Q_B say how good it is: target = r + γ · Q_B(s′, argmax_a Q_A(s′, a)). The lucky noise in Q_A is not repeated in Q_B, so the target is not systematically too high.
# Q-learning: the same table chooses AND evaluates
target = r + GAMMA * max(QA[s2])
# Double Q-learning: QA chooses, QB evaluates (and swap roles at random)
b = argmax(QA[s2])
target = r + GAMMA * QB[s2][b]
3. Stability traps in DQN
| Trap | What happens | Usual guard |
|---|---|---|
| Learning rate too big | weights jump; Q-values swing; policy flips | smaller step size |
| No error clipping | one surprising transition kicks the weights far | clip the error to ±1 (Huber-style) |
| Target synced too often | moving target, like no target network at all | sync every C updates (or a soft update) |
| Overestimation | Q-values drift above true returns; bad actions look good | Double Q / Double DQN |
| Small, stale replay buffer | network over-fits recent experience | bigger buffer, warm-up before training |
| One seed | you believe a lucky (or unlucky) run | always test several seeds |
Visual
Visualise it: targets, syncs and TD error
Online estimator, replay buffer and a lagged target you sync by hand. Watch how the TD error behaves when the target moves too often. (Educational simulation.)
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Code lab
Code lab: catch the optimist
overestimation_lab.py has three parts. Part 1 averages max() of noisy estimates (20,000 tries). Part 2 is a gripper decision: from START, GO HOME (reward 0, end) or TRY SHELF, then pick one of 10 grasps with reward drawn from a normal distribution with mean −0.1 and sd 1 (so the shelf is worse on average). Q-learning and Double Q-learning (α 0.1, ε 0.1, 300 episodes) are each run with 500 seeds. Part 3 reuses the tiny DQN from Lesson 8 (150 episodes, 3 seeds) with deliberately bad settings.
"""overestimation_lab.py - Stage 14 L09: stability traps and overestimation.
Part 1: the max of noisy guesses is too high, even when every true value is 0.
Part 2: a gripper decision - Q-learning vs Double Q-learning.
From START the robot can GO HOME (episode ends, reward 0) or TRY SHELF.
At the shelf it picks one of 10 grasp angles; every grasp is a bit worse than
going home on average (reward ~ N(-0.1, 1)), then the episode ends.
The best policy is GO HOME. Which learner discovers that?
Part 3: stability traps in a tiny DQN (servo-hold task from L08): learning rate and target sync.
Standard library only.
"""
import math
import random
print("1) True value of every action = 0. Each estimate = true value + noise (sd 1).")
print(" actions average of max(estimates) (should be 0)")
rng = random.Random(0)
for n in (1, 2, 5, 10, 50):
avg = sum(max(rng.gauss(0, 1) for _ in range(n)) for _ in range(20000)) / 20000
print(f" {n:7d} {avg:+.2f}")
def shelf_task(double, seed, episodes=300, n_grasps=10):
rng = random.Random(seed)
QA = {"start": [0.0, 0.0], "shelf": [0.0] * n_grasps} # start: 0 HOME, 1 SHELF
QB = {"start": [0.0, 0.0], "shelf": [0.0] * n_grasps}
alpha, eps, shelf_choices = 0.1, 0.1, []
def act(key, n):
if rng.random() < eps:
return rng.randrange(n)
q = [a + b for a, b in zip(QA[key], QB[key])] if double else QA[key]
best = max(q)
return rng.choice([i for i, v in enumerate(q) if v == best])
for ep in range(episodes):
a = act("start", 2)
shelf_choices.append(a == 1)
if a == 0: # GO HOME: reward 0, done
if double and rng.random() < 0.5:
QB["start"][0] += alpha * (0 - QB["start"][0])
else:
QA["start"][0] += alpha * (0 - QA["start"][0])
continue
# TRY SHELF: reward 0, next state = shelf
if double:
Q1, Q2 = (QA, QB) if rng.random() < 0.5 else (QB, QA)
b = max(range(n_grasps), key=lambda i: Q1["shelf"][i]) # choose with one table
Q1["start"][1] += alpha * (0 + Q2["shelf"][b] - Q1["start"][1]) # value with the other
else:
QA["start"][1] += alpha * (0 + max(QA["shelf"]) - QA["start"][1])
g = act("shelf", n_grasps)
r = rng.gauss(-0.1, 1.0)
if double:
T = QA if rng.random() < 0.5 else QB
T["shelf"][g] += alpha * (r - T["shelf"][g])
else:
QA["shelf"][g] += alpha * (r - QA["shelf"][g])
return shelf_choices
print("\n2) Shelf task: % of episodes where the robot chose TRY SHELF (best = only exploration, ~5 %)")
print(" episodes Q-learning Double Q-learning (average of 500 seeds)")
runs_q = [shelf_task(False, s) for s in range(500)]
runs_d = [shelf_task(True, s) for s in range(500)]
for lo, hi in ((0, 10), (10, 50), (50, 100), (100, 200), (200, 300)):
pq = sum(sum(r[lo:hi]) for r in runs_q) / (500 * (hi - lo)) * 100
pd = sum(sum(r[lo:hi]) for r in runs_d) / (500 * (hi - lo)) * 100
print(f" {lo + 1:3d}-{hi:<3d} {pq:5.1f} % {pd:5.1f} %")
# ---------- Part 3: tiny DQN stability traps ----------
H, NA, GAMMA = 16, 3, 0.9
def new_net(rng):
return {"w1": [rng.gauss(0, 1.0) for _ in range(H)], "b1": [0.0] * H,
"w2": [[rng.gauss(0, 0.3) for _ in range(H)] for _ in range(NA)], "b2": [0.0] * NA}
def copy_net(n):
return {"w1": n["w1"][:], "b1": n["b1"][:], "w2": [r[:] for r in n["w2"]], "b2": n["b2"][:]}
def forward(n, x):
h = [math.tanh(n["w1"][j] * x + n["b1"][j]) for j in range(H)]
return [sum(n["w2"][a][j] * h[j] for j in range(H)) + n["b2"][a] for a in range(NA)], h
def train_step(n, x, a, target, lr, clip):
q, h = forward(n, x)
err = q[a] - target
if clip:
err = max(-1.0, min(1.0, err))
for j in range(H):
g_h = err * n["w2"][a][j] * (1 - h[j] ** 2)
n["w2"][a][j] -= lr * err * h[j]
n["w1"][j] -= lr * g_h * x
n["b1"][j] -= lr * g_h
n["b2"][a] -= lr * err
def dqn(seed, lr=0.01, sync=200, clip=True, episodes=150):
rng = random.Random(seed)
net = new_net(rng)
tgt, buf, updates = copy_net(net), [], 0
for ep in range(episodes):
x, eps = rng.uniform(0, 180), max(0.05, 1 - ep / (episodes * 0.6))
for _ in range(40):
q, _ = forward(net, x / 90 - 1)
a = rng.randrange(NA) if rng.random() < eps else max(range(NA), key=lambda i: q[i])
x2 = min(180.0, max(0.0, x + (a - 1) * 5))
buf.append((x / 90 - 1, a, -abs(x2 - 90) / 10, x2 / 90 - 1))
buf = buf[-5000:]
for s, aa, r, s2 in rng.sample(buf, min(16, len(buf))):
train_step(net, s, aa, r + GAMMA * max(forward(tgt, s2)[0]), lr, clip)
updates += 1
if updates % sync == 0:
tgt = copy_net(net)
x = x2
errs, qmax = [], 0.0
for _ in range(50):
x = rng.uniform(0, 180)
for _ in range(40):
q, _ = forward(net, x / 90 - 1)
qmax = max(qmax, max(abs(v) for v in q))
x = min(180.0, max(0.0, x + (max(range(NA), key=lambda i: q[i]) - 1) * 5))
errs.append(abs(x - 90))
if any(math.isnan(e) for e in errs) or math.isnan(qmax):
return 0.0, float("nan")
return sum(1 for e in errs if e <= 5) / 50, qmax
print("\n3) Tiny DQN stability traps (servo-hold, 150 episodes, 3 seeds; true |Q| is at most about 50)")
print(" setting within 5 deg (per seed) biggest |Q| seen")
for name, kw in (("baseline lr 0.01, sync 200, clip", {}),
("lr 0.1 (10x too big)", {"lr": 0.1}),
("lr 0.1, no error clipping", {"lr": 0.1, "clip": False}),
("sync every update (no freeze)", {"sync": 1}),
("sync every 5000 (rare sync)", {"sync": 5000})):
res = [dqn(s, **kw) for s in range(3)]
oks = " / ".join(f"{r[0] * 100:3.0f} %" for r in res)
qm = max(r[1] for r in res)
print(f" {name:36s} {oks:26s} {qm:10.1f}")Output (run with python3 overestimation_lab.py)
1) True value of every action = 0. Each estimate = true value + noise (sd 1).
actions average of max(estimates) (should be 0)
1 +0.00
2 +0.57
5 +1.16
10 +1.55
50 +2.24
2) Shelf task: % of episodes where the robot chose TRY SHELF (best = only exploration, ~5 %)
episodes Q-learning Double Q-learning (average of 500 seeds)
1-10 68.8 % 48.4 %
11-50 89.2 % 23.9 %
51-100 65.7 % 11.7 %
101-200 27.1 % 9.0 %
201-300 12.5 % 6.9 %
3) Tiny DQN stability traps (servo-hold, 150 episodes, 3 seeds; true |Q| is at most about 50)
setting within 5 deg (per seed) biggest |Q| seen
baseline lr 0.01, sync 200, clip 92 % / 30 % / 0 % 51.0
lr 0.1 (10x too big) 14 % / 0 % / 10 % 57.4
lr 0.1, no error clipping 14 % / 0 % / 0 % 28.8
sync every update (no freeze) 92 % / 0 % / 0 % 48.6
sync every 5000 (rare sync) 100 % / 100 % / 94 % 50.7Try it: change the sync interval and train many batches. Compare the TD-error sparkline when the target is synced very often vs rarely. (Educational simulation.)
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
Bench build: overestimation from a real sensor
noisy_max_ldr.ino needs only the LDR divider. With the light and sensor still, every “action” has the same true value (the average light level). Each estimate is the average of only 3 readings, so it is noisy. The sketch reports the honest mean, the single max (choose and value with the same estimates) and the double estimate (choose with set A, value with set B), averaged over 200 trials with 8 actions.
| Part | Connection | Note |
|---|---|---|
| LDR | 5 V → LDR → A0 | the same divider as Lesson 2 |
| 10 kΩ resistor | A0 → GND | sets the divider |
| Light source | steady desk lamp; try a flickering tube light too | more noise = more overestimation |

// noisy_max_ldr.ino - Stage 14 L09
// Overestimation with a REAL sensor. Point the LDR at a steady light and do not move anything.
// We pretend there are N "actions" whose true values are all the same: the average light level.
// Each action's estimate = average of a few noisy LDR readings (like a Q-value from few samples).
// single max : pick the best estimate and report ITS value (what Q-learning does)
// double : pick the best with estimate set A, report set B's value for that action
// The single max is too high on average; the double estimate is not.
//
// Wiring: LDR 5V -> A0, 10k A0 -> GND (same divider as L02). No motors. Serial 115200.
// Tip: a flickering LED lamp or tube light gives bigger noise - and bigger overestimation.
const int LDR_PIN = A0, N_ACTIONS = 8, SAMPLES = 3, TRIALS = 200;
float noisyEstimate() { // a "Q-value" from only a few samples
long s = 0;
for (int i = 0; i < SAMPLES; i++) { s += analogRead(LDR_PIN); delayMicroseconds(700); }
return s / (float)SAMPLES;
}
void setup() {
Serial.begin(115200);
delay(500);
Serial.println(F("# keep the light and sensor still"));
}
void loop() {
double sumMean = 0, sumSingle = 0, sumDouble = 0;
for (int t = 0; t < TRIALS; t++) {
float A[N_ACTIONS], B[N_ACTIONS];
for (int a = 0; a < N_ACTIONS; a++) { A[a] = noisyEstimate(); B[a] = noisyEstimate(); }
int best = 0;
float mean = 0;
for (int a = 0; a < N_ACTIONS; a++) { if (A[a] > A[best]) best = a; mean += A[a]; }
sumMean += mean / N_ACTIONS; // the honest average (true value estimate)
sumSingle += A[best]; // max of the same estimates: biased upward
sumDouble += B[best]; // chosen by A, valued by B: unbiased
}
Serial.print(F("true (mean) ")); Serial.print(sumMean / TRIALS, 2);
Serial.print(F(" | single max ")); Serial.print(sumSingle / TRIALS, 2);
Serial.print(F(" (+")); Serial.print((sumSingle - sumMean) / TRIALS, 2);
Serial.print(F(") | double ")); Serial.print(sumDouble / TRIALS, 2);
Serial.print(F(" (")); Serial.print((sumDouble - sumMean) / TRIALS, 2);
Serial.println(F(")"));
}# keep the light and sensor still
true (mean) <value> | single max <higher value> (+<bias>) | double <about the mean> (<small +/->)
true (mean) <value> | single max <higher value> (+<bias>) | double <about the mean> (<small +/->)
...Debugging
| Symptom | Likely cause | Fix |
|---|---|---|
| Single max shows no bias | the reading is not noisy (very bright or saturated) | Dim the light or use a tube light; check the reading is mid-range |
| Readings drift up or down over time | the light really changes (sun, people moving) | Shade the sensor; use a lamp, not daylight |
| Double Q still picks the bad option sometimes | ε exploration and noise | That is expected — compare averages over many seeds |
| Q-values in DQN keep climbing above real returns | overestimation | Use Double DQN targets; check against greedy test returns |
| Training collapses after looking good | learning rate or sync interval | Lower LR; test several C values |
| Python Part 3 slow | pure-Python network | Run fewer seeds while experimenting |
Real world
Overestimation in the real world

- Double DQN: the same choose-with-one, value-with-the-other idea added to DQN, which outperforms the original DQN (Wikipedia, Q-learning).
- Robots with many actions: more discrete actions = bigger max bias, which is one reason to keep action lists short and meaningful.
- Auctions and selection: the winner’s curse — the most optimistic bidder wins and tends to overpay — is the same statistics outside robotics.
- Engineering habit: whenever you pick “the best” from noisy measurements (best grasp, best gain, best model), re-measure it on fresh data before trusting its number.
Think
Think about it
Think
A robot chooses between 20 grasp angles, each tried only twice. It reports “best grasp success 95 %”. Should you believe 95 %? How would you check it?
Experiment
Try it in the lab
About 65 minutes. Python 3 for the three parts; an Arduino UNO with an LDR for the bench test.
Experiment
Measure the optimism
You will need
- `overestimation_lab.py` (Python 3, no libraries)
- DQN intuition lab (in your browser)
- Arduino UNO, LDR + 10 kΩ, jumper wires
- Desk lamp (and a tube light if you have one)
Steps
- Run `overestimation_lab.py` and copy all three parts into your notebook.
- In Part 2, change `n_grasps` from 10 to 2 and to 50. How does Q-learning’s shelf % in episodes 11–50 change?
- In Part 2, change the grasp reward mean from −0.1 to +0.1 (now the shelf is better). Do both learners find it?
- In Part 3, try sync 50, 1000 and 20000. Plot sync interval vs success.
- Upload `noisy_max_ldr.ino`. Record 10 lines with a steady lamp, then 10 with a flickering light. Compare the bias.
- Change SAMPLES to 30 and N_ACTIONS to 2 (separately). Record the bias each time.
Interactive · DQN intuition lab · Educational value-based deep RL toy
Table cells
16
Q(s,a) entries
Net params
4
linear φ · 2 feats × 2 acts
|TD| mean
0.000
last train batch
Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0
Online Q-net (updating)
← [0.00, 0.00] → [0.00, 0.00]
Target Q-net (lagged) · sync in 8
← [0.00, 0.00] → [0.00, 0.00]
Replay buffer (0/24) · highlight = last sampled batch
|TD error| sparkline
Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.
φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.
max() over noisy estimates is biased upward, more so with more actions and more noise. Double estimation removes the bias by separating choosing from valuing. DQN training has many traps; error clipping, a sensible learning rate and a well-tested target sync interval keep it stable.
Knowledge check
Check your understanding
Knowledge check
1. All actions are truly worth 0 but estimates are noisy. The max of the estimates is on average…
2. The overestimation bias gets bigger when…
3. Double Q-learning target is…
4. In Part 2, why did Q-learning go to the shelf so often early on?
5. Error clipping in DQN…
6. Syncing the target network after every update is like…
7. In Part 3, syncing every 5000 updates did best. The lesson is…
8. Your robot reports very high Q-values. The best check is…
Mini assignment
Hands-on assignment
Mini assignment
Overestimation investigation
Show overestimation in simulation and on real hardware, and show a fix.
Deliverables
- A graph from Part 1 with 1–50 actions and two noise levels (sd 0.5 and 1).
- Part 2 results for 2, 10 and 50 grasps, Q vs Double Q, with a short explanation.
- Your Part 3 sync-interval graph, and the setting you would pick, with reasons.
- Bench table: bias (single max and double) for a steady and a flickering light, with SAMPLES 3 and 30.
Do this, step by step
- Run
overestimation_lab.pyPart 1 for 1–50 actions at sd 0.5 and 1; plot the bias. - Run Part 2 for 2, 10 and 50 grasps with Q and Double Q.
- Run Part 3 sync settings and pick one, with a reason.
- Bench: upload
noisy_max_ldr.inoand log 10 lines with a steady lamp, then 10 with a flicker. - Change
SAMPLESfrom 3 to 30 and repeat both.
Stuck? Hints, one at a time
- Single-max bias grows with the number of actions and with noise — more choices, more chances for a lucky high estimate.
- Double estimates choose with set A and value with set B, so the bias drops near 0.
- “sync every update” means no freeze at all — expect the wobble DQN’s target network is there to stop.
Challenges (pick by level)
- 🟢 Beginner: Ask 10 friends to guess the number of sweets in a jar. Is the highest guess usually too high? Relate it to max().
- 🟡 Intermediate: Add “average Q at START” to Part 2’s print-out for both learners, and compare it with the true value (0 for home).
- 🟠 Advanced: Add a Double DQN target to Part 3: choose the next action with the online network, value it with the target network. Compare the biggest |Q| seen.
- 🔴 Expert: In
noisy_max_ldr.ino, compute the bias for N_ACTIONS = 1…16 and print a table. Fit it against the Part 1 numbers scaled by your sensor’s noise. - ⚫ Research: Read the Double Q-learning section of Wikipedia’s Q-learning article and write how Double DQN reuses the target network as the “second estimate”.
Next step
You now know the full DQN story: why tables break, how a function generalises, how replay and target networks stabilise it, and how overestimation sneaks in. In Lesson 10 — DQN Intuition Challenge you will put it all together on one design challenge for a lab robot.
Look back: Lesson 8: DQN Core Ideas. Next: Lesson 10: DQN Intuition Challenge. Related: Stage 12 — Simulation & Digital Twin.
Sources
- Wikipedia — Q-learning (Double Q-learning and Deep Q-learning sections) — overestimation in noisy environments because the same Q selects and evaluates; Double Q-learning update with Q_A and Q_B trained on separate experiences; Double DQN outperforms the original DQN.
- Wikipedia — Winner’s curse — the most optimistic bidder wins and tends to overpay.
- OpenAI Baselines — DQN code (GitHub) — reference DQN implementation using the Huber loss (error clipping).
- Mnih et al. (2015) — Human-level control through deep reinforcement learning (PDF, DeepMind) — target network and replay for stability (Lesson 8).
- Photos and diagrams: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).