Stage 14 · Lesson 07/10

Advanced Robotics / Reinforcement Learning · Lesson 07

Function Approximation Intuition

Est. 65–80 min

Why

Why this lesson matters

In Lesson 6 the table broke in two ways: it grew as bins^variables, and each box had to be learnt on its own. Function approximation fixes both. Instead of one stored number per (state, action), we keep a small set of weights w and compute Q(s, a; w) from the state. Change a weight and the estimate changes for a whole neighbourhood of states at once — that is generalisation.

Today you will see one update spread to nearby angles, beat the 36-, 180- and 900-bin tables on the same servo task with just 33 weights, learn how the choice of features decides what the robot can learn, and run a linear Q-learner on the light-seeker with 18 weights. This is the exact idea a DQN (Lesson 8) uses — with a neural network as the formula.

Concept

From boxes to a formula

1. Table vs function

Wikipedia’s Q-learning article: combining Q-learning with function approximation makes larger and continuous state spaces possible, and can speed up learning because the algorithm generalises earlier experience to unseen states.
Q-tableFunction approximation
What is storedone number per (state, action)a fixed set of weights w
Sizegrows as bins^variables × actionschosen by you (tens to millions of weights)
Learning at 40°changes only the 40° boxalso changes 41°, 45°, 50°… (less and less with distance)
Unseen statestill at its starting valuegets a sensible guess from its neighbours
Riskslow, but each box is exactone update can also spoil a far-away guess
A plot of seven red data points at x = 0 to 6 with gaps between them
A table knows only the points it has visited. Everything between the dots is empty.Photo: Berland · Public domain · Source: Wikimedia Commons · resized, converted to WebP
A smooth curve drawn through ten black data points
A function fills the gaps with a smooth guess. Function approximation in RL does the same with Q-values: learn at a few states, guess sensibly everywhere in between.Photo: Soul windsurfer · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

2. Linear Q with features

The simplest approximator is linear: turn the state into a list of features f(s) (numbers that describe it), and keep one weight per feature per action. Then Q(s, a) = w_a · f(s) = w_a,1·f₁(s) + w_a,2·f₂(s) + … For a servo angle, a good feature set is a few bumps (radial basis functions, RBFs): each bump is 1 when the angle is at its centre and fades to 0 further away. A radial basis function is a function whose value depends only on the distance from a fixed centre point.

Four bell-shaped curves of different widths and centres on one graph
Bell curves of different widths. Our bump features look like these: a narrow bump only “sees” angles very close to its centre; a wide bump spreads learning over a big range.Photo: Inductiveload · Public domain · Source: Wikimedia Commons · resized, converted to WebP
Features and the update (from function_approx_lab.py) · Python
def rbf_features(x, centres, width):
    return [math.exp(-((x - c) / width) ** 2) for c in centres] + [1.0]   # bumps + constant

# Q(x, a) = sum(W[a][i] * f[i])
err = r + 0.9 * max(Q(x2, b) for b in actions) - Q(x, a)     # same TD error as Q-learning
step = 0.3 / sum(fi * fi for fi in f)                        # normalised step size
W[a] = [wi + step * err * fi for wi, fi in zip(W[a], f)]     # every weight moves by its feature

3. Features decide what can be learnt

Too few or too wide bumps blur everything together — the robot cannot tell 80° from 100°. Too narrow bumps stop sharing — you are back to a table with gaps between the boxes. Good features match how fast the true values change with the state. Choosing them by hand is hard for images or many joints; a neural network learns its own features (Lesson 8).

Visual

Visualise it: one set of weights, every state

A tiny Q-network: one set of weights produces Q-values for every state. Train on a few transitions and watch unvisited states change too. (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Code lab

Code lab: one formula beats three tables

function_approx_lab.py uses the same servo task as Lesson 6: move to 90° and hold (actions −5°, 0, +5°; reward −|error|/10; 300 episodes, ε 1 → 0.05, 5 seeds). Part 1 makes a single update at 40° and prints what changed. Part 2 compares tables (36/180/900 bins) with linear Q on 10 RBF bumps. Part 3 tries different bump counts and widths. Standard library only.

function_approx_lab.py · Python
"""function_approx_lab.py - Stage 14 L07: function approximation intuition.

Replace the Q-table with a formula: Q(s, a; w) = w_a . features(s).
Part 1: generalisation - one update at 40 deg also changes nearby angles (table: only its own bin).
Part 2: the L06 servo task (hold 90 deg) - tables with 36/180/900 bins vs linear Q with
        radial-basis features, same 300 episodes, 5 seeds.
Part 3: too few or too narrow features - the approximation trade-off.
Standard library only.
"""
import math
import random


def rbf_features(x, centres, width):
    """Bumps of 'closeness' to each centre angle (0..1), plus a constant 1."""
    return [math.exp(-((x - c) / width) ** 2) for c in centres] + [1.0]


def make_centres(n):
    return [i * 180 / (n - 1) for i in range(n)]


print("1) One update at 40 deg: what changes?")
centres, width = make_centres(10), 20.0
w = [0.0] * 11
f40 = rbf_features(40, centres, width)
target, alpha = 1.0, 0.5
pred = sum(wi * fi for wi, fi in zip(w, f40))
w = [wi + alpha * (target - pred) * fi / sum(f * f for f in f40) for wi, fi in zip(w, f40)]
print("   angle   table (bin 5 deg)   linear RBF")
for x in (40, 41, 45, 50, 60, 90, 150):
    table = 0.5 if int(x / 5) == int(40 / 5) else 0.0
    lin = sum(wi * fi for wi, fi in zip(w, rbf_features(x, centres, width)))
    print(f"   {x:4d}    {table:8.2f}            {lin:6.2f}")
print("   (table learning rate 0.5, both start at 0; the target at 40 deg was 1.0)")


def servo_episode_loop(choose, update, episodes, rng):
    for ep in range(episodes):
        x, eps = rng.uniform(0, 180), max(0.05, 1 - ep / (episodes * 0.6))
        for _ in range(40):
            a = choose(x, eps)
            x2 = min(180.0, max(0.0, x + (a - 1) * 5))
            r = -abs(x2 - 90) / 10
            update(x, a, r, x2)
            x = x2


def test(greedy, rng):
    errs = []
    for _ in range(50):
        x = rng.uniform(0, 180)
        for _ in range(40):
            x = min(180.0, max(0.0, x + (greedy(x) - 1) * 5))
        errs.append(abs(x - 90))
    return sum(1 for e in errs if e <= 5) / 50, sum(errs) / 50


def run_table(bins, seed):
    rng = random.Random(seed)
    Q = {}
    width = 180 / bins
    b = lambda x: min(bins - 1, int(x / width))
    def q(x):
        return Q.setdefault(b(x), [0.0] * 3)
    def choose(x, eps):
        v = q(x)
        return rng.randrange(3) if rng.random() < eps else max(range(3), key=lambda i: (v[i], rng.random()))
    def update(x, a, r, x2):
        v = q(x)
        v[a] += 0.3 * (r + 0.9 * max(q(x2)) - v[a])
    servo_episode_loop(choose, update, 300, rng)
    return test(lambda x: max(range(3), key=lambda i: Q.get(b(x), [0.0] * 3)[i]), rng), bins * 3


def run_linear(n, width, seed):
    rng = random.Random(seed)
    centres = make_centres(n)
    W = [[0.0] * (n + 1) for _ in range(3)]
    def qv(x):
        f = rbf_features(x, centres, width)
        return [sum(wi * fi for wi, fi in zip(W[a], f)) for a in range(3)], f
    def choose(x, eps):
        v, _ = qv(x)
        return rng.randrange(3) if rng.random() < eps else max(range(3), key=lambda i: (v[i], rng.random()))
    def update(x, a, r, x2):
        v, f = qv(x)
        v2, _ = qv(x2)
        err = r + 0.9 * max(v2) - v[a]
        step = 0.3 / sum(fi * fi for fi in f)          # normalised step size
        W[a] = [wi + step * err * fi for wi, fi in zip(W[a], f)]
    servo_episode_loop(choose, update, 300, rng)
    return test(lambda x: max(range(3), key=lambda i: qv(x)[0][i]), rng), 3 * (n + 1)


def summary(results):
    ok = sum(r[0][0] for r in results) / len(results)
    err = sum(r[0][1] for r in results) / len(results)
    return ok, err, results[0][1]


print("\n2) Hold 90 deg: table vs linear function (300 episodes, 5 seeds)")
print("   method                         numbers learnt   ends within 5 deg   avg final error")
for bins in (36, 180, 900):
    ok, err, n = summary([run_table(bins, s) for s in range(5)])
    print(f"   table, {bins:3d} bins                 {n:6d}            {ok * 100:5.0f} %          {err:6.1f} deg")
for n, wd in ((10, 20.0),):
    ok, err, k = summary([run_linear(n, wd, s) for s in range(5)])
    print(f"   linear, {n} RBF bumps (w {wd:.0f} deg)    {k:6d}            {ok * 100:5.0f} %          {err:6.1f} deg")

print("\n3) Choosing features (linear RBF, 300 episodes, 5 seeds)")
print("   bumps  width   weights   ends within 5 deg   avg final error")
for n, wd in ((3, 60.0), (5, 40.0), (10, 20.0), (19, 10.0), (19, 2.0)):
    ok, err, k = summary([run_linear(n, wd, s) for s in range(5)])
    print(f"   {n:5d}  {wd:4.0f}    {k:6d}       {ok * 100:5.0f} %             {err:6.1f} deg")

Output (run with python3 function_approx_lab.py)

1) One update at 40 deg: what changes?
   angle   table (bin 5 deg)   linear RBF
     40        0.50              0.50
     41        0.50              0.50
     45        0.00              0.49
     50        0.00              0.46
     60        0.00              0.39
     90        0.00              0.23
    150        0.00              0.22
   (table learning rate 0.5, both start at 0; the target at 40 deg was 1.0)

2) Hold 90 deg: table vs linear function (300 episodes, 5 seeds)
   method                         numbers learnt   ends within 5 deg   avg final error
   table,  36 bins                    108               89 %            11.3 deg
   table, 180 bins                    540               37 %            37.8 deg
   table, 900 bins                   2700               11 %            45.1 deg
   linear, 10 RBF bumps (w 20 deg)        33               96 %             2.9 deg

3) Choosing features (linear RBF, 300 episodes, 5 seeds)
   bumps  width   weights   ends within 5 deg   avg final error
       3    60        12          91 %                2.6 deg
       5    40        18          75 %                5.5 deg
      10    20        33          96 %                2.9 deg
      19    10        60          98 %                2.4 deg
      19     2        60           8 %               42.9 deg

Try it: the network inside is a function approximator. Collect transitions and train — then look at the Q-values for states you have not visited. (Educational simulation.)

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

Bench build: a light-seeker with 18 weights

linear_q_light_seeker.ino is the Lesson 4 light-seeker with the table replaced by linear Q. The head moves in 10° steps between 30° and 150°, and episodes start at any of 25 angles (5° apart). Five bumps at 30, 60, 90, 120 and 150° (width 30°) plus a constant give 6 features × 3 actions = 18 weights, where a 25-position table would need 75. Every 10 episodes it prints Q and the greedy action at every 5° — including angles where the head never stopped.

Safety: 60 episodes is a lot of servo movement — use a separate servo supply (never a GPIO pin or the UNO 5 V pin), keep fingers away from the horn, and let the servo rest if it gets warm.
PartConnectionNote
SG90/MG90-class servosignal → D9the same light-seeker head
Servo supplyred → separate 5–6 V, brown → supply GNDjoin supply GND to UNO GND
LDR + 10 kΩLDR 5 V → A0, 10 kΩ A0 → GNDcalibrate LEVEL_EDGES (Lesson 2)
Push buttonD2 → GND (internal pull-up)pause / resume
linear_q_light_seeker.ino · Arduino C++
// linear_q_light_seeker.ino - Stage 14 L07
// The L04 light-seeker without a table: Q(angle, a) = w[a] . features(angle).
// State  = servo angle 30..150 deg in 5 deg steps (25 positions - finer than L04's 9 bins)
// Actions: 0 LEFT, 1 STAY, 2 RIGHT (10 deg).  Features: 5 bumps centred at 30, 60, 90, 120,
// 150 deg (width 30 deg) + a constant 1  ->  only 3 x 6 = 18 weights instead of 25 x 3 = 75.
// Reward as L04: -0.1 per step, +1 brighter, -1 darker, +5 and TERMINATED at light level 3.
//
// Wiring: same as L01-L05. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND, button D2 -> GND = pause / resume. Serial 115200.
// Never power the servo from a GPIO pin or the UNO 5 V pin.
#include <Servo.h>

const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int NF = 6, NA = 3, MAX_STEPS = 25, EPISODES = 60;
const int LEVEL_EDGES[3] = {300, 550, 800};       // calibrate for your room (see L02)
const float ALPHA = 0.2, GAMMA = 0.9;
const float CENTRES[5] = {30, 60, 90, 120, 150}, WIDTH = 30;

Servo head;
float W[NA][NF];                                   // 18 floats = 72 bytes, all start at 0
int angle = 90, episode = 0;
bool paused = false;

void features(int x, float f[NF]) {
  for (int i = 0; i < 5; i++) { float d = (x - CENTRES[i]) / WIDTH; f[i] = exp(-d * d); }
  f[5] = 1.0;                                      // constant feature (bias)
}

float qValue(int x, int a) {
  float f[NF], q = 0;
  features(x, f);
  for (int i = 0; i < NF; i++) q += W[a][i] * f[i];
  return q;
}

int bestAction(int x) {
  float q[NA];
  for (int a = 0; a < NA; a++) q[a] = qValue(x, a);
  int best = 0;
  for (int a = 1; a < NA; a++) if (q[a] > q[best]) best = a;
  int ties[NA], n = 0;
  for (int a = 0; a < NA; a++) if (q[a] == q[best]) ties[n++] = a;
  return ties[random(n)];
}

float maxQ(int x) { return max(qValue(x, 0), max(qValue(x, 1), qValue(x, 2))); }

int readLevel() {
  long sum = 0;
  for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
  int raw = sum / 8, level = 0;
  while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
  return level;
}

void moveTo(int to) {
  int from = head.read(), dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
  head.write(to);
  delay(300);
}

void checkPause() {
  if (digitalRead(BUTTON_PIN) == LOW) {
    delay(30);
    while (digitalRead(BUTTON_PIN) == LOW) {}
    paused = !paused;
    Serial.println(paused ? F("# paused") : F("# resumed"));
  }
}

void runEpisode() {
  float eps = max(0.05, 1.0 - episode / 30.0);
  angle = 30 + 5 * random(25);
  moveTo(angle);
  int level = readLevel(), t = 0;
  float g = 0;
  bool terminated = false;
  while (t < MAX_STEPS && !terminated) {
    int x = angle, a = (random(1000) < eps * 1000) ? random(NA) : bestAction(x);
    int x2 = constrain(x + (a - 1) * 10, 30, 150);
    if (x2 != x) moveTo(x2);
    int level2 = readLevel();
    float r = -0.1;
    if (level2 > level) r = 1.0; else if (level2 < level) r = -1.0;
    if (level2 == 3) { r = 5.0; terminated = true; }
    float target = terminated ? r : r + GAMMA * maxQ(x2);
    float err = target - qValue(x, a), f[NF], ff = 0;
    features(x, f);
    for (int i = 0; i < NF; i++) ff += f[i] * f[i];
    for (int i = 0; i < NF; i++) W[a][i] += (ALPHA / ff) * err * f[i];   // semi-gradient update
    angle = x2; level = level2; g += r; t++;
  }
  episode++;
  Serial.print(episode); Serial.print(','); Serial.print(eps, 2); Serial.print(',');
  Serial.print(t); Serial.print(','); Serial.print(g, 1); Serial.print(',');
  Serial.println(terminated ? F("terminated") : F("truncated"));
}

void printPolicy() {   // every 5 deg - including angles the robot never stood at
  Serial.println(F("# angle: Q[LEFT] Q[STAY] Q[RIGHT] -> greedy"));
  for (int x = 30; x <= 150; x += 5) {
    Serial.print(x); Serial.print(": ");
    for (int a = 0; a < NA; a++) { Serial.print(qValue(x, a), 2); Serial.print(' '); }
    int b = bestAction(x);
    Serial.println(b == 0 ? F("-> LEFT") : (b == 1 ? F("-> STAY") : F("-> RIGHT")));
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  randomSeed(analogRead(A5));
  head.attach(SERVO_PIN);
  head.write(90);
  delay(500);
  Serial.println(F("episode,epsilon,steps,return,end"));
}

void loop() {
  checkPause();
  if (paused) return;
  if (episode < EPISODES) {
    runEpisode();
    if (episode % 10 == 0) printPolicy();
  }
}
Serial Monitor (115200) — shape of the output (your numbers will differ) · Text
episode,epsilon,steps,return,end
1,1.00,<steps>,<return>,<terminated/truncated>
...
# angle: Q[LEFT] Q[STAY] Q[RIGHT] -> greedy
30: <q> <q> <q> -> RIGHT
35: <q> <q> <q> -> RIGHT
...
150: <q> <q> <q> -> LEFT

Debugging

SymptomLikely causeFix
Q-values grow huge or become nanstep size too big for the featuresUse the normalised step (α / Σf²) or a smaller α
Same greedy action at every anglebumps too wide (or only the constant is learning)More or narrower bumps
Good at visited angles, random in betweenbumps too narrow — no sharingWiden the bumps (about the spacing between centres)
Learning undoes itself when the torch movesgeneralisation spreads new errorsSmaller α; more varied starting angles
Python numbers differ from the tabledifferent seed or episode countKeep seeds 0–4 and 300 episodes to compare
UNO resets when the servo movesservo on UNO 5 VSeparate supply, grounds joined

Real world

Function approximation in the real world

An old contour map of hills with fine brown contour lines and streams
A contour map stores a whole landscape with a few lines; you estimate the height anywhere between them. A value function over a robot’s states is like such a landscape — function approximation stores its shape, not every point.Photo: W. H. Otis (donated by Hobart M. Lovett) · Public domain · Source: Wikimedia Commons · resized, converted to WebP
  • Continuous control: joint angles, speeds and forces go straight into the features or network — no bins.
  • Balancing tasks: Wikipedia’s Q-learning article describes balancing a stick on a finger: four numbers (finger position, velocity, stick angle, angular velocity) give infinitely many states, so they must be quantised or approximated.
  • Robots from pixels: hand-made bumps do not work for images; neural networks learn their own features. That is the DQN in Lesson 8.
  • Embedded robots: linear Q with a few dozen weights fits easily in an UNO — useful for small learning gadgets in a school lab.

Think

Think about it

Think

You have only one feature: a constant 1. What will linear Q learn about the light-seeker, and what will its greedy policy look like?

Experiment

Try it in the lab

About 65 minutes. Python 3 for the feature tests; the light-seeker hardware for the bench run.

Experiment

Tune the features

You will need

  • `function_approx_lab.py` (Python 3, no libraries)
  • DQN intuition lab (in your browser)
  • Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
  • LED torch on a stand

Steps

  1. Run `function_approx_lab.py`. Copy all three parts into your notebook.
  2. In Part 1, remove the constant feature (the `+ [1.0]`). What happens to 150° now?
  3. In Part 3, add (19, 20.0) and (40, 5.0). Draw a graph of width vs “within 5°” for 19 bumps.
  4. Change the target from 90° to 37°, which is not a bump centre. Does linear Q still beat the tables?
  5. On the bench, run the 60 episodes, then copy the final policy. Mark the angles where the head never stopped.
  6. Move the torch to the other side and run 20 more episodes (reset the episode counter). How quickly does the policy flip?

Interactive · DQN intuition lab · Educational value-based deep RL toy

Browser toy only — NOT MuJoCo / Isaac / stable-baselines / production DQN. Tiny linear Q approx + replay + lagged target for intuition.

Table cells

16

Q(s,a) entries

Net params

4

linear φ · 2 feats × 2 acts

|TD| mean

0.000

last train batch

→●
→
→
→
→
→
→
G

Lane · arrows = greedy from online Q · start left · G = goal · α=0.25 γ=0.9 · steps=0

Online Q-net (updating)

← [0.00, 0.00]
→ [0.00, 0.00]

Target Q-net (lagged) · sync in 8

← [0.00, 0.00]
→ [0.00, 0.00]

Replay buffer (0/24) · highlight = last sampled batch

|TD error| sparkline

Train to populate

Collect with ε-greedy → store in replay → sample batch → update online Q → sync target.

φ(s)=[pos_norm, dist_to_goal] · Q(s,a)=w_a·φ · target bootstraps TD · replay breaks step correlation — intuition only.

A small formula can beat a big table because it shares learning between nearby states. The features decide what it can and cannot tell apart: wide = blurry, narrow = gappy. Generalisation helps unseen states but also spreads errors.

Knowledge check

Check your understanding

Knowledge check

1. Function approximation in RL means…

2. Generalisation means…

3. In linear Q, Q(s, a) =

4. In Part 2, linear Q with 33 weights held 90° within 5° in…

5. Why did 19 bumps of width 2° do badly (8 %)?

6. A Q-table is a special case of linear function approximation where…

7. A risk of function approximation is…

8. For camera images, instead of hand-made bump features we usually…

Mini assignment

Hands-on assignment

Mini assignment

Feature design report

Design, test and explain features for a learning robot in your lab.

Deliverables

  • Your Part 3 graph (bump count and width vs “within 5°”) with at least 7 settings, and the setting you would choose, with reasons.
  • The Part 1 test with and without the constant feature, and one sentence about what the constant does.
  • The final bench policy table for the 18-weight light-seeker, marking unvisited angles, and the torch-moved re-learning result.
  • A one-page plan: which features you would use for a 2-joint arm reaching a target (angles, distances, …) and why.

Do this, step by step

  1. Run function_approx_lab.py Part 1 with and without the constant feature.
  2. In Part 3, try at least 7 settings of bump count and width; graph “within 5°”.
  3. Bench: upload linear_q_light_seeker.ino (18 weights) and run all 60 episodes.
  4. Mark which printed angles the head never visited, then move the torch and watch it re-learn.
  5. Write your feature plan for a 2-joint reaching arm.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Bumps that are too narrow act like a table again; too wide and every angle looks the same.
  2. Without the constant feature, Q is forced near 0 far from all bumps.
  3. If the weights blow up, lower ALPHA (0.2) — big features plus a big step overshoot.

Challenges (pick by level)

  • 🟢 Beginner: Explain generalisation using a cricket example: a batter who has faced 140 km/h balls can guess how to play a 142 km/h ball. What would a “table batter” do?
  • 🟡 Intermediate: Replace the RBF bumps in Part 2 with polynomial features (1, x, x², x³ with x scaled to 0..1). Compare the results.
  • 🟠 Advanced: Use 2-D bumps for a 2-joint arm (a grid of centres over both angles). Count the weights and compare with a 10×10 table.
  • 🔴 Expert: Implement tile coding (several shifted bin grids, one-hot each) as the feature set and compare with RBFs at the same number of weights.
  • ⚫ Research: Read Sutton & Barto Part II (Ch. 9–10). Write half a page on why a semi-gradient update is “not a true gradient”.

Next step

Hand-made features work for one or two angles, but not for a camera image. In Lesson 8 — DQN Core Ideas the formula becomes a neural network that learns its own features, and you will meet the two tricks that made it work: experience replay and a target network.

Look back: Lesson 6: Why Tabular Q Breaks. Next: Lesson 8: DQN Core Ideas. Related: Stage 12 — Simulation & Digital Twin.

Sources

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in