Stage 14 · Lesson 09/10

Advanced Robotics / Reinforcement Learning · Lesson 09

Stability Traps & Overestimation

Est. 65–80 min

Why

Why this lesson matters

Q-learning’s target uses max over Q(s′, ·). When the Q-values are noisy estimates, the max tends to pick an action whose estimate happened to come out too high. So targets are biased upward — overestimation. Wikipedia’s Q-learning article puts it this way: because the same Q function both selects and evaluates the next action, Q-learning can overestimate action values in noisy environments, slowing learning. Double Q-learning fixes this by selecting with one estimate and evaluating with another.

Today you will measure the max bias with numbers, watch Q-learning get fooled on a gripper decision while Double Q-learning is not, probe the classic DQN stability traps (learning rate, clipping, target sync), and see real overestimation from your own LDR’s noise on the bench.

Concept

Why the max lies, and how training goes wrong

1. The max of noisy guesses is too high

Suppose all actions are truly worth 0, but each estimate has random noise. The max of the estimates is almost always above 0 — and the more actions, the bigger the error. In Part 1 below (noise with standard deviation 1): 2 actions → +0.57, 10 actions → +1.55, 50 actions → +2.24, even though the true best value is 0.

A crowded fish auction at a dock with buyers in colourful saris and shirts, one holding a basket overhead
A fish auction at Sassoon Docks, Mumbai. In auctions where the item is worth the same to everyone, the winner is the bidder with the most optimistic estimate and tends to overpay — the “winner’s curse”. max() over noisy Q-values does the same.Photo: Rudolph.A.furtado · CC0 · Source: Wikimedia Commons · resized, converted to WebP

2. Double Q-learning: choose with one, value with the other

Keep two estimates, Q_A and Q_B, each trained on different experiences (flip a coin each step). To update Q_A, let Q_A choose the best next action, but let Q_B say how good it is: target = r + γ · Q_B(s′, argmax_a Q_A(s′, a)). The lucky noise in Q_A is not repeated in Q_B, so the target is not systematically too high.

The one-line difference (from overestimation_lab.py) · Python
# Q-learning: the same table chooses AND evaluates
target = r + GAMMA * max(QA[s2])

# Double Q-learning: QA chooses, QB evaluates (and swap roles at random)
b = argmax(QA[s2])
target = r + GAMMA * QB[s2][b]
A hand with a red pen marking numbers on a lottery ticket
Picking the “luckiest” numbers does not make them more likely to win. A value picked because it looked best on noisy data needs a fresh, independent check.Photo: Santeri Viinamäki · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

3. Stability traps in DQN

Wikipedia’s Q-learning article notes that Double Q-learning was later combined with DQN, as Double DQN, which outperforms the original DQN algorithm.
TrapWhat happensUsual guard
Learning rate too bigweights jump; Q-values swing; policy flipssmaller step size
No error clippingone surprising transition kicks the weights farclip the error to ±1 (Huber-style)
Target synced too oftenmoving target, like no target network at allsync every C updates (or a soft update)
OverestimationQ-values drift above true returns; bad actions look goodDouble Q / Double DQN
Small, stale replay buffernetwork over-fits recent experiencebigger buffer, warm-up before training
One seedyou believe a lucky (or unlucky) runalways test several seeds

Visual

Visualise it: targets, syncs and TD error

Online estimator, replay buffer and a lagged target you sync by hand. Watch how the TD error behaves when the target moves too often. (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Code lab

Code lab: catch the optimist

overestimation_lab.py has three parts. Part 1 averages max() of noisy estimates (20,000 tries). Part 2 is a gripper decision: from START, GO HOME (reward 0, end) or TRY SHELF, then pick one of 10 grasps with reward drawn from a normal distribution with mean −0.1 and sd 1 (so the shelf is worse on average). Q-learning and Double Q-learning (α 0.1, ε 0.1, 300 episodes) are each run with 500 seeds. Part 3 reuses the tiny DQN from Lesson 8 (150 episodes, 3 seeds) with deliberately bad settings.

overestimation_lab.py · Python
"""overestimation_lab.py - Stage 14 L09: stability traps and overestimation.

Part 1: the max of noisy guesses is too high, even when every true value is 0.
Part 2: a gripper decision - Q-learning vs Double Q-learning.
        From START the robot can GO HOME (episode ends, reward 0) or TRY SHELF.
        At the shelf it picks one of 10 grasp angles; every grasp is a bit worse than
        going home on average (reward ~ N(-0.1, 1)), then the episode ends.
        The best policy is GO HOME. Which learner discovers that?
Part 3: stability traps in a tiny DQN (servo-hold task from L08): learning rate and target sync.
Standard library only.
"""
import math
import random

print("1) True value of every action = 0. Each estimate = true value + noise (sd 1).")
print("   actions   average of max(estimates)   (should be 0)")
rng = random.Random(0)
for n in (1, 2, 5, 10, 50):
    avg = sum(max(rng.gauss(0, 1) for _ in range(n)) for _ in range(20000)) / 20000
    print(f"   {n:7d}   {avg:+.2f}")


def shelf_task(double, seed, episodes=300, n_grasps=10):
    rng = random.Random(seed)
    QA = {"start": [0.0, 0.0], "shelf": [0.0] * n_grasps}   # start: 0 HOME, 1 SHELF
    QB = {"start": [0.0, 0.0], "shelf": [0.0] * n_grasps}
    alpha, eps, shelf_choices = 0.1, 0.1, []

    def act(key, n):
        if rng.random() < eps:
            return rng.randrange(n)
        q = [a + b for a, b in zip(QA[key], QB[key])] if double else QA[key]
        best = max(q)
        return rng.choice([i for i, v in enumerate(q) if v == best])

    for ep in range(episodes):
        a = act("start", 2)
        shelf_choices.append(a == 1)
        if a == 0:                                        # GO HOME: reward 0, done
            if double and rng.random() < 0.5:
                QB["start"][0] += alpha * (0 - QB["start"][0])
            else:
                QA["start"][0] += alpha * (0 - QA["start"][0])
            continue
        # TRY SHELF: reward 0, next state = shelf
        if double:
            Q1, Q2 = (QA, QB) if rng.random() < 0.5 else (QB, QA)
            b = max(range(n_grasps), key=lambda i: Q1["shelf"][i])   # choose with one table
            Q1["start"][1] += alpha * (0 + Q2["shelf"][b] - Q1["start"][1])   # value with the other
        else:
            QA["start"][1] += alpha * (0 + max(QA["shelf"]) - QA["start"][1])
        g = act("shelf", n_grasps)
        r = rng.gauss(-0.1, 1.0)
        if double:
            T = QA if rng.random() < 0.5 else QB
            T["shelf"][g] += alpha * (r - T["shelf"][g])
        else:
            QA["shelf"][g] += alpha * (r - QA["shelf"][g])
    return shelf_choices


print("\n2) Shelf task: % of episodes where the robot chose TRY SHELF (best = only exploration, ~5 %)")
print("   episodes      Q-learning   Double Q-learning     (average of 500 seeds)")
runs_q = [shelf_task(False, s) for s in range(500)]
runs_d = [shelf_task(True, s) for s in range(500)]
for lo, hi in ((0, 10), (10, 50), (50, 100), (100, 200), (200, 300)):
    pq = sum(sum(r[lo:hi]) for r in runs_q) / (500 * (hi - lo)) * 100
    pd = sum(sum(r[lo:hi]) for r in runs_d) / (500 * (hi - lo)) * 100
    print(f"   {lo + 1:3d}-{hi:<3d}         {pq:5.1f} %          {pd:5.1f} %")

# ---------- Part 3: tiny DQN stability traps ----------
H, NA, GAMMA = 16, 3, 0.9


def new_net(rng):
    return {"w1": [rng.gauss(0, 1.0) for _ in range(H)], "b1": [0.0] * H,
            "w2": [[rng.gauss(0, 0.3) for _ in range(H)] for _ in range(NA)], "b2": [0.0] * NA}


def copy_net(n):
    return {"w1": n["w1"][:], "b1": n["b1"][:], "w2": [r[:] for r in n["w2"]], "b2": n["b2"][:]}


def forward(n, x):
    h = [math.tanh(n["w1"][j] * x + n["b1"][j]) for j in range(H)]
    return [sum(n["w2"][a][j] * h[j] for j in range(H)) + n["b2"][a] for a in range(NA)], h


def train_step(n, x, a, target, lr, clip):
    q, h = forward(n, x)
    err = q[a] - target
    if clip:
        err = max(-1.0, min(1.0, err))
    for j in range(H):
        g_h = err * n["w2"][a][j] * (1 - h[j] ** 2)
        n["w2"][a][j] -= lr * err * h[j]
        n["w1"][j] -= lr * g_h * x
        n["b1"][j] -= lr * g_h
    n["b2"][a] -= lr * err


def dqn(seed, lr=0.01, sync=200, clip=True, episodes=150):
    rng = random.Random(seed)
    net = new_net(rng)
    tgt, buf, updates = copy_net(net), [], 0
    for ep in range(episodes):
        x, eps = rng.uniform(0, 180), max(0.05, 1 - ep / (episodes * 0.6))
        for _ in range(40):
            q, _ = forward(net, x / 90 - 1)
            a = rng.randrange(NA) if rng.random() < eps else max(range(NA), key=lambda i: q[i])
            x2 = min(180.0, max(0.0, x + (a - 1) * 5))
            buf.append((x / 90 - 1, a, -abs(x2 - 90) / 10, x2 / 90 - 1))
            buf = buf[-5000:]
            for s, aa, r, s2 in rng.sample(buf, min(16, len(buf))):
                train_step(net, s, aa, r + GAMMA * max(forward(tgt, s2)[0]), lr, clip)
                updates += 1
                if updates % sync == 0:
                    tgt = copy_net(net)
            x = x2
    errs, qmax = [], 0.0
    for _ in range(50):
        x = rng.uniform(0, 180)
        for _ in range(40):
            q, _ = forward(net, x / 90 - 1)
            qmax = max(qmax, max(abs(v) for v in q))
            x = min(180.0, max(0.0, x + (max(range(NA), key=lambda i: q[i]) - 1) * 5))
        errs.append(abs(x - 90))
    if any(math.isnan(e) for e in errs) or math.isnan(qmax):
        return 0.0, float("nan")
    return sum(1 for e in errs if e <= 5) / 50, qmax


print("\n3) Tiny DQN stability traps (servo-hold, 150 episodes, 3 seeds; true |Q| is at most about 50)")
print("   setting                              within 5 deg (per seed)     biggest |Q| seen")
for name, kw in (("baseline lr 0.01, sync 200, clip", {}),
                 ("lr 0.1 (10x too big)", {"lr": 0.1}),
                 ("lr 0.1, no error clipping", {"lr": 0.1, "clip": False}),
                 ("sync every update (no freeze)", {"sync": 1}),
                 ("sync every 5000 (rare sync)", {"sync": 5000})):
    res = [dqn(s, **kw) for s in range(3)]
    oks = " / ".join(f"{r[0] * 100:3.0f} %" for r in res)
    qm = max(r[1] for r in res)
    print(f"   {name:36s} {oks:26s}  {qm:10.1f}")

Output (run with python3 overestimation_lab.py)

1) True value of every action = 0. Each estimate = true value + noise (sd 1).
   actions   average of max(estimates)   (should be 0)
         1   +0.00
         2   +0.57
         5   +1.16
        10   +1.55
        50   +2.24

2) Shelf task: % of episodes where the robot chose TRY SHELF (best = only exploration, ~5 %)
   episodes      Q-learning   Double Q-learning     (average of 500 seeds)
     1-10           68.8 %           48.4 %
    11-50           89.2 %           23.9 %
    51-100          65.7 %           11.7 %
   101-200          27.1 %            9.0 %
   201-300          12.5 %            6.9 %

3) Tiny DQN stability traps (servo-hold, 150 episodes, 3 seeds; true |Q| is at most about 50)
   setting                              within 5 deg (per seed)     biggest |Q| seen
   baseline lr 0.01, sync 200, clip      92 % /  30 % /   0 %             51.0
   lr 0.1 (10x too big)                  14 % /   0 % /  10 %             57.4
   lr 0.1, no error clipping             14 % /   0 % /   0 %             28.8
   sync every update (no freeze)         92 % /   0 % /   0 %             48.6
   sync every 5000 (rare sync)          100 % / 100 % /  94 %             50.7

Try it: change the sync interval and train many batches. Compare the TD-error sparkline when the target is synced very often vs rarely. (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Bench build: overestimation from a real sensor

noisy_max_ldr.ino needs only the LDR divider. With the light and sensor still, every “action” has the same true value (the average light level). Each estimate is the average of only 3 readings, so it is noisy. The sketch reports the honest mean, the single max (choose and value with the same estimates) and the double estimate (choose with set A, value with set B), averaged over 200 trials with 8 actions.

Safety: no motors in this sketch. If you power a lamp for the test, use its own mains or USB supply — never a GPIO pin. Keep mains wiring away from the breadboard.
PartConnectionNote
LDR5 V → LDR → A0the same divider as Lesson 2
10 kΩ resistorA0 → GNDsets the divider
Light sourcesteady desk lamp; try a flickering tube light toomore noise = more overestimation
A round light-dependent resistor with an orange zig-zag track and two wire legs
A light-dependent resistor (LDR). Its reading wobbles a little even under a steady lamp — perfect free noise for an overestimation experiment.Photo: Suyash Dwivedi · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP
noisy_max_ldr.ino · Arduino C++
// noisy_max_ldr.ino - Stage 14 L09
// Overestimation with a REAL sensor. Point the LDR at a steady light and do not move anything.
// We pretend there are N "actions" whose true values are all the same: the average light level.
// Each action's estimate = average of a few noisy LDR readings (like a Q-value from few samples).
//   single max  : pick the best estimate and report ITS value       (what Q-learning does)
//   double      : pick the best with estimate set A, report set B's value for that action
// The single max is too high on average; the double estimate is not.
//
// Wiring: LDR 5V -> A0, 10k A0 -> GND (same divider as L02). No motors. Serial 115200.
// Tip: a flickering LED lamp or tube light gives bigger noise - and bigger overestimation.

const int LDR_PIN = A0, N_ACTIONS = 8, SAMPLES = 3, TRIALS = 200;

float noisyEstimate() {                 // a "Q-value" from only a few samples
  long s = 0;
  for (int i = 0; i < SAMPLES; i++) { s += analogRead(LDR_PIN); delayMicroseconds(700); }
  return s / (float)SAMPLES;
}

void setup() {
  Serial.begin(115200);
  delay(500);
  Serial.println(F("# keep the light and sensor still"));
}

void loop() {
  double sumMean = 0, sumSingle = 0, sumDouble = 0;
  for (int t = 0; t < TRIALS; t++) {
    float A[N_ACTIONS], B[N_ACTIONS];
    for (int a = 0; a < N_ACTIONS; a++) { A[a] = noisyEstimate(); B[a] = noisyEstimate(); }
    int best = 0;
    float mean = 0;
    for (int a = 0; a < N_ACTIONS; a++) { if (A[a] > A[best]) best = a; mean += A[a]; }
    sumMean += mean / N_ACTIONS;        // the honest average (true value estimate)
    sumSingle += A[best];               // max of the same estimates: biased upward
    sumDouble += B[best];               // chosen by A, valued by B: unbiased
  }
  Serial.print(F("true (mean) ")); Serial.print(sumMean / TRIALS, 2);
  Serial.print(F(" | single max ")); Serial.print(sumSingle / TRIALS, 2);
  Serial.print(F(" (+")); Serial.print((sumSingle - sumMean) / TRIALS, 2);
  Serial.print(F(") | double ")); Serial.print(sumDouble / TRIALS, 2);
  Serial.print(F(" (")); Serial.print((sumDouble - sumMean) / TRIALS, 2);
  Serial.println(F(")"));
}
Serial Monitor (115200) — shape of the output (your numbers will differ) · Text
# keep the light and sensor still
true (mean) <value> | single max <higher value> (+<bias>) | double <about the mean> (<small +/->)
true (mean) <value> | single max <higher value> (+<bias>) | double <about the mean> (<small +/->)
...

Debugging

SymptomLikely causeFix
Single max shows no biasthe reading is not noisy (very bright or saturated)Dim the light or use a tube light; check the reading is mid-range
Readings drift up or down over timethe light really changes (sun, people moving)Shade the sensor; use a lamp, not daylight
Double Q still picks the bad option sometimesε exploration and noiseThat is expected — compare averages over many seeds
Q-values in DQN keep climbing above real returnsoverestimationUse Double DQN targets; check against greedy test returns
Training collapses after looking goodlearning rate or sync intervalLower LR; test several C values
Python Part 3 slowpure-Python networkRun fewer seeds while experimenting

Real world

Overestimation in the real world

Four bell-shaped curves of different widths on one graph
Noise in estimates is like these bell curves: the wider the curve (more noise) and the more actions you compare, the further the max drifts above the truth.Photo: Inductiveload · Public domain · Source: Wikimedia Commons · resized, converted to WebP
  • Double DQN: the same choose-with-one, value-with-the-other idea added to DQN, which outperforms the original DQN (Wikipedia, Q-learning).
  • Robots with many actions: more discrete actions = bigger max bias, which is one reason to keep action lists short and meaningful.
  • Auctions and selection: the winner’s curse — the most optimistic bidder wins and tends to overpay — is the same statistics outside robotics.
  • Engineering habit: whenever you pick “the best” from noisy measurements (best grasp, best gain, best model), re-measure it on fresh data before trusting its number.

Think

Think about it

Think

A robot chooses between 20 grasp angles, each tried only twice. It reports “best grasp success 95 %”. Should you believe 95 %? How would you check it?

Experiment

Try it in the lab

About 65 minutes. Python 3 for the three parts; an Arduino UNO with an LDR for the bench test.

Experiment

Measure the optimism

You will need

  • `overestimation_lab.py` (Python 3, no libraries)
  • DQN intuition lab (in your browser)
  • Arduino UNO, LDR + 10 kΩ, jumper wires
  • Desk lamp (and a tube light if you have one)

Steps

  1. Run `overestimation_lab.py` and copy all three parts into your notebook.
  2. In Part 2, change `n_grasps` from 10 to 2 and to 50. How does Q-learning’s shelf % in episodes 11–50 change?
  3. In Part 2, change the grasp reward mean from −0.1 to +0.1 (now the shelf is better). Do both learners find it?
  4. In Part 3, try sync 50, 1000 and 20000. Plot sync interval vs success.
  5. Upload `noisy_max_ldr.ino`. Record 10 lines with a steady lamp, then 10 with a flickering light. Compare the bias.
  6. Change SAMPLES to 30 and N_ACTIONS to 2 (separately). Record the bias each time.

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

max() over noisy estimates is biased upward, more so with more actions and more noise. Double estimation removes the bias by separating choosing from valuing. DQN training has many traps; error clipping, a sensible learning rate and a well-tested target sync interval keep it stable.

Knowledge check

Check your understanding

Knowledge check

1. All actions are truly worth 0 but estimates are noisy. The max of the estimates is on average…

2. The overestimation bias gets bigger when…

3. Double Q-learning target is…

4. In Part 2, why did Q-learning go to the shelf so often early on?

5. Error clipping in DQN…

6. Syncing the target network after every update is like…

7. In Part 3, syncing every 5000 updates did best. The lesson is…

8. Your robot reports very high Q-values. The best check is…

Mini assignment

Hands-on assignment

Mini assignment

Overestimation investigation

Show overestimation in simulation and on real hardware, and show a fix.

Deliverables

  • A graph from Part 1 with 1–50 actions and two noise levels (sd 0.5 and 1).
  • Part 2 results for 2, 10 and 50 grasps, Q vs Double Q, with a short explanation.
  • Your Part 3 sync-interval graph, and the setting you would pick, with reasons.
  • Bench table: bias (single max and double) for a steady and a flickering light, with SAMPLES 3 and 30.

Do this, step by step

  1. Run overestimation_lab.py Part 1 for 1–50 actions at sd 0.5 and 1; plot the bias.
  2. Run Part 2 for 2, 10 and 50 grasps with Q and Double Q.
  3. Run Part 3 sync settings and pick one, with a reason.
  4. Bench: upload noisy_max_ldr.ino and log 10 lines with a steady lamp, then 10 with a flicker.
  5. Change SAMPLES from 3 to 30 and repeat both.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Single-max bias grows with the number of actions and with noise — more choices, more chances for a lucky high estimate.
  2. Double estimates choose with set A and value with set B, so the bias drops near 0.
  3. “sync every update” means no freeze at all — expect the wobble DQN’s target network is there to stop.

Challenges (pick by level)

  • 🟢 Beginner: Ask 10 friends to guess the number of sweets in a jar. Is the highest guess usually too high? Relate it to max().
  • 🟡 Intermediate: Add “average Q at START” to Part 2’s print-out for both learners, and compare it with the true value (0 for home).
  • 🟠 Advanced: Add a Double DQN target to Part 3: choose the next action with the online network, value it with the target network. Compare the biggest |Q| seen.
  • 🔴 Expert: In noisy_max_ldr.ino, compute the bias for N_ACTIONS = 1…16 and print a table. Fit it against the Part 1 numbers scaled by your sensor’s noise.
  • ⚫ Research: Read the Double Q-learning section of Wikipedia’s Q-learning article and write how Double DQN reuses the target network as the “second estimate”.

Next step

You now know the full DQN story: why tables break, how a function generalises, how replay and target networks stabilise it, and how overestimation sneaks in. In Lesson 10 — DQN Intuition Challenge you will put it all together on one design challenge for a lab robot.

Look back: Lesson 8: DQN Core Ideas. Next: Lesson 10: DQN Intuition Challenge. Related: Stage 12 — Simulation & Digital Twin.

Sources

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in