Stage 14 · Lesson 05/10

Advanced Robotics / Reinforcement Learning · Lesson 05

Reward Shaping for Manipulation

Est. 70–85 min

Why

Why this lesson matters

For manipulation, the true goal is usually sparse: “+10 when the block is in the bin”, nothing else. A robot exploring randomly may take hundreds of episodes to stumble on that reward even once. Reward shaping adds extra in-between rewards to guide it — but a careless bonus changes what the robot learns to want. This is called reward hacking or specification gaming.

Today you will train the same Q-learner on a pick-and-place rail with six different training rewards, judge every result on the *true* task, watch two robots hack their rewards, and learn the one form of shaping that is mathematically safe: potential-based shaping (Ng, Harada & Russell, 1999). On the bench, the light-seeker will compare sparse, naive and potential-based rewards live.

Concept

Sparse, dense and shaped rewards

1. Sparse vs dense rewards

The Gymnasium custom-environment guide lists “only rewarding at the very end” as a common pitfall and suggests a small step penalty or a distance-based term as better options.
TypeExample for pick-and-placeGoodBad
Sparse+10 only when the block lands in the binsays exactly what you want; hard to hackvery slow to learn — the robot rarely sees any reward
Step costsparse, plus −0.1 per stepencourages finishing quicklystill sparse at the start
Dense / shapedsparse, plus bonuses for progress (getting closer, grasping)fast learning signaleasy to create loopholes the robot exploits
A small dog in a blue checked sweater sitting up and looking at a treat held in a person’s fingers
Dog trainers reward small steps towards a trick. Robots learn from rewards in a similar way — and, like a clever dog, a robot will repeat whatever earns the treat, whether or not that is what you meant.Photo: Pete Bellis petebellis · CC0 · Source: Wikimedia Commons · resized, converted to WebP

2. Reward hacking: the robot finds a shortcut

Reward hacking happens when an agent maximises the reward as written without achieving what the designer intended. Famous examples on Wikipedia: a bicycle-riding agent rewarded for moving towards the goal (but not punished for moving away) learnt to ride in tiny circles; a 2004 Lego Mindstorms robot rewarded for staying on a path learnt to zig-zag backwards over the straight part; a DeepMind arm flipped a brick because the grasp reward used the wrong reference point on the brick.

A man in a yellow shirt riding a bicycle with a large container tied to the back, under a flyover on a city road
A real rider goes where the job needs him to go. A learning agent rewarded only for “moving towards the goal” found it could earn more by circling forever — every half-circle points towards the goal for a moment.Photo: Fquasie · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

3. Potential-based shaping: the safe way

Ng, Harada and Russell proved in 1999 that if the extra reward has the form F(s, a, s′) = γ·Φ(s′) − Φ(s) for some “potential” function Φ over states, then the optimal policy does not change. And without further knowledge of the MDP, this is the *only* kind of shaping with that guarantee. Φ can be any sensible “how close am I to done” score — for example minus the distance to the next subgoal.

Potential-based shaping for pick-and-place · Python
def potential(s):
    gripper, block = s
    if block == "H":                         # holding: get to the bin
        return 10 - abs(BIN - gripper)
    return -abs(block - gripper) - abs(BIN - block)   # go to the block first

shaped_r = true_r + GAMMA * potential(s2) - potential(s)

Visual

Visualise it: three reward modes

Switch between sparse, dense and naive shaped rewards on a toy pick episode — and try to “wiggle” your way to a high return without picking anything. (Educational simulation.)

Interactive · Reward shaping lab · Educational

Browser toy only — NOT MuJoCo / Isaac / stable-baselines reward APIs. Demonstrates sparse vs dense vs naive shaping failure modes.

Only +1 on successful pick, 0 otherwise. Honest but hard to learn from few trials.

Phase

idle

Episode return

0.00

Hack risk (toy)

10%

toy pick cell · EE → object

Choose a reward mode, then step a toy pick episode.

Code lab

Code lab: six training rewards, one true task

reward_shaping_lab.py puts a gripper on a 10-cell rail with the block at cell 2 and the bin at cell 8. Actions: LEFT, RIGHT, GRASP, RELEASE. The true reward is +10 for releasing the block into the bin. Tabular Q-learning (α 0.3, γ 0.95, ε 1 → 0.05) trains for 600 episodes with each of six training rewards and 5 seeds; every result is then measured with a greedy test on the true task.

reward_shaping_lab.py · Python
"""reward_shaping_lab.py - Stage 14 L05: shaping the reward for a pick-and-place task.

A gripper slides along a 10-cell rail. A block starts at cell 2, the bin is at cell 8.
State  : (gripper cell 0..9, block cell 0..9 or 'H' = held)
Actions: 0 LEFT, 1 RIGHT, 2 GRASP, 3 RELEASE
TRUE task reward (what we really want): +10 when the block is released into the bin,
episode ends. Nothing else. Max 60 steps.

The robot learns with tabular Q-learning (alpha 0.3, gamma 0.95, epsilon 1 -> 0.05)
under four different TRAINING rewards. We always judge it on the TRUE task.
Standard library only.
"""
import random

N, BIN, BLOCK0, MAX_STEPS, GAMMA = 10, 8, 2, 60, 0.95
LEFT, RIGHT, GRASP, RELEASE = range(4)


def step(s, a):
    g, b = s
    if a == LEFT:
        g = max(0, g - 1)
    elif a == RIGHT:
        g = min(N - 1, g + 1)
    elif a == GRASP and b == g:
        b = "H"
    elif a == RELEASE and b == "H":
        if g == BIN:
            return (g, BIN), True, True          # placed in bin: success
        b = g                                    # dropped on the rail
    return (g, b), False, False


def potential(s):
    """Phi(s): higher = closer to finishing. Distance-to-subgoal, plus a step for holding."""
    g, b = s
    if b == "H":
        return 10 - abs(BIN - g)                 # carrying: go to the bin
    return -abs(b - g) - abs(BIN - b)            # not carrying: go to the block, then it must travel


def train_reward(kind, s, a, s2, success, grasped):
    r = 10.0 if success else 0.0
    if kind == "sparse":
        return r
    if kind == "step cost":
        return r - 0.1
    if kind == "+2 per grasp":
        return r + (2.0 if grasped else 0.0)                    # careless bonus
    if kind in ("+1 when closer", "+3 when closer"):
        closer = potential(s2) > potential(s)
        bonus = 1.0 if kind == "+1 when closer" else 3.0
        return r + (bonus if closer else 0.0)                   # rewards progress, never punishes going back
    if kind == "potential-based":
        return r + GAMMA * potential(s2) - potential(s)         # Ng, Harada & Russell 1999
    raise ValueError(kind)


def choose(Q, s, eps, rng):
    q = Q.setdefault(s, [0.0] * 4)
    if rng.random() < eps:
        return rng.randrange(4)
    m = max(q)
    return rng.choice([a for a in range(4) if q[a] == m])


def train(kind, episodes=600, seed=0):
    rng, Q, first_success = random.Random(seed), {}, None
    for ep in range(episodes):
        s, eps = (rng.randrange(N), BLOCK0), max(0.05, 1 - ep / 300)
        for _ in range(MAX_STEPS):
            a = choose(Q, s, eps, rng)
            s2, success, done = step(s, a)
            grasped = s[1] != "H" and s2[1] == "H"
            r = train_reward(kind, s, a, s2, success, grasped)
            q2 = 0.0 if done else max(Q.setdefault(s2, [0.0] * 4))
            Q[s][a] += 0.3 * (r + GAMMA * q2 - Q[s][a])
            s = s2
            if done:
                if first_success is None:
                    first_success = ep + 1
                break
    return Q, first_success


def test(Q, episodes=200, seed=123):
    """Greedy test on the TRUE task: success rate, steps when successful, grasps per episode."""
    rng = random.Random(seed)
    wins, steps_won, grasps = 0, 0, 0
    for _ in range(episodes):
        s = (rng.randrange(N), BLOCK0)
        for t in range(MAX_STEPS):
            a = choose(Q, s, 0.0, rng)
            s2, success, done = step(s, a)
            grasps += s[1] != "H" and s2[1] == "H"
            s = s2
            if done:
                wins += 1
                steps_won += t + 1
                break
    return wins / episodes, (steps_won / wins if wins else float("nan")), grasps / episodes


def main():
    kinds = ["sparse", "step cost", "+2 per grasp", "+1 when closer", "+3 when closer", "potential-based"]
    print("Pick-and-place on a 10-cell rail (block at 2, bin at 8). 600 training episodes x 5 seeds.")
    print("Every result below is measured on the TRUE task (+10 only for block in bin).\n")
    print(f"   {'training reward':17s} {'first success (episode)':>26s} {'greedy success':>15s} {'steps':>6s} {'grasps/ep':>10s}")
    for kind in kinds:
        firsts, succ, steps, gr = [], 0.0, [], 0.0
        for seed in range(5):
            Q, first = train(kind, seed=seed)
            sr, st, g = test(Q, seed=200 + seed)
            firsts.append(first)
            succ += sr / 5
            gr += g / 5
            if sr > 0:
                steps.append(st)
        avg_steps = sum(steps) / len(steps) if steps else float("nan")
        print(f"   {kind:17s} {str(firsts):>26s} {succ * 100:13.0f} % {avg_steps:6.1f} {gr:10.2f}")

    print("\nWhat does the '+2 per grasp' robot actually do? (greedy, seed 0, start at cell 0)")
    Q, _ = train("+2 per grasp", seed=0)
    s, names, trace = (0, BLOCK0), "LRGD", []
    for _ in range(24):
        a = choose(Q, s, 0.0, random.Random(1))
        trace.append(names[a])
        s, _, done = step(s, a)
        if done:
            trace.append("| DONE")
            break
    print("   " + " ".join(trace))
    print("   (L/R = move, G = grasp, D = release/drop)")

    print("\nWhat does the '+3 when closer' robot do? (greedy, seed 0, start at cell 0)")
    Q, _ = train("+3 when closer", seed=0)
    s, trace = (0, BLOCK0), []
    for _ in range(24):
        a = choose(Q, s, 0.0, random.Random(1))
        trace.append(names[a])
        s, _, done = step(s, a)
        if done:
            trace.append("| DONE")
            break
    print("   " + " ".join(trace))

    print("\nWhy loops cannot be farmed with potential-based shaping (gamma = 1 for clarity):")
    a, b = (2, BLOCK0), (3, BLOCK0)       # gripper on the block, then one step away
    closer = lambda x, y: 3.0 if potential(y) > potential(x) else 0.0
    print(f"   '+3 when closer' loop A->B->A earns {closer(a, b) + closer(b, a):+.1f} every 2 steps")
    print(f"   potential-based  loop A->B->A earns {(potential(b) - potential(a)) + (potential(a) - potential(b)):+.1f}"
          "  (the bonuses telescope: only the start and end state matter)")


if __name__ == "__main__":
    main()

Output (run with python3 reward_shaping_lab.py)

Pick-and-place on a 10-cell rail (block at 2, bin at 8). 600 training episodes x 5 seeds.
Every result below is measured on the TRUE task (+10 only for block in bin).

   training reward      first success (episode)  greedy success  steps  grasps/ep
   sparse              [149, 72, 125, None, 46]            80 %   18.7       1.06
   step cost               [72, 41, 58, 55, 56]           100 %   11.1       1.00
   +2 per grasp      [None, 67, None, None, 44]             0 %    nan      28.66
   +1 when closer       [139, 66, 101, 141, 63]           100 %   11.1       1.00
   +3 when closer       [139, 66, 101, 141, 63]             0 %    nan       1.00
   potential-based         [47, 41, 40, 35, 33]           100 %   11.1       1.00

What does the '+2 per grasp' robot actually do? (greedy, seed 0, start at cell 0)
   R R G D G D G D G D G D G D G D G D G D G D G D
   (L/R = move, G = grasp, D = release/drop)

What does the '+3 when closer' robot do? (greedy, seed 0, start at cell 0)
   R R G R R R R R R L R L R L R L R L R L R L R L

Why loops cannot be farmed with potential-based shaping (gamma = 1 for clarity):
   '+3 when closer' loop A->B->A earns +3.0 every 2 steps
   potential-based  loop A->B->A earns +0.0  (the bonuses telescope: only the start and end state matter)

Try it: choose Sparse, Dense or Naive shaped and step a toy pick episode. Under Naive shaped, try Wiggle and watch the return grow without any pick. (Educational simulation.)

Interactive · Reward shaping lab · Educational

Browser toy only — NOT MuJoCo / Isaac / stable-baselines reward APIs. Demonstrates sparse vs dense vs naive shaping failure modes.

Only +1 on successful pick, 0 otherwise. Honest but hard to learn from few trials.

Phase

idle

Episode return

0.00

Hack risk (toy)

10%

toy pick cell · EE → object

Choose a reward mode, then step a toy pick episode.

Bench build: compare rewards on the light-seeker

shaping_compare.ino is the Lesson 4 Q-learner with a choice of training reward, typed into the Serial Monitor: s = sparse (+5 at light level 3 only), n = naive (+1 whenever the light gets brighter, nothing for darker), p = potential-based (γ·level′ − level). Each run clears the Q-table and trains 40 episodes, logging steps, success and the number of back-and-forth moves.

Safety: three runs of 40 episodes is a lot of servo movement — use a separate servo supply (never a GPIO pin or the UNO 5 V pin), keep fingers away from the horn, and let the servo rest if it gets warm.
PartConnectionNote
SG90/MG90-class servosignal → D9the same light-seeker head
Servo supplyred → separate 5–6 V, brown → supply GNDjoin supply GND to UNO GND
LDR + 10 kΩLDR 5 V → A0, 10 kΩ A0 → GNDcalibrate LEVEL_EDGES (Lesson 2)
shaping_compare.ino · Arduino C++
// shaping_compare.ino - Stage 14 L05
// Same Q-learning light-seeker as L04, but you choose the TRAINING reward from the Serial Monitor:
//   's' = sparse        : +5 only when light level 3 is reached (episode terminates)
//   'n' = naive bonus   : sparse + 1 whenever the light level goes UP (nothing for going down)
//   'p' = potential     : sparse + (GAMMA * phi(s') - phi(s)), phi = light level  (Ng et al. 1999)
// Each choice clears the Q-table and trains 40 episodes. The log counts "back-and-forth" moves
// (LEFT right after RIGHT or the reverse) - the tell-tale sign of a robot farming a bonus.
//
// Wiring: same as L01-L04. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND. Serial 115200, line ending "No line ending" or "Newline".
#include <Servo.h>

const int SERVO_PIN = 9, LDR_PIN = A0;
const int N = 9, NA = 3, MAX_STEPS = 25, EPISODES = 40;
const int LEVEL_EDGES[3] = {300, 550, 800};       // calibrate for your room (see L02)
const float ALPHA = 0.3, GAMMA = 0.9;

Servo head;
float Q[N][NA];
char mode = 0;
int angleBin = 4;

int binToAngle(int b) { return 30 + 15 * b; }

int readLevel() {
  long sum = 0;
  for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
  int raw = sum / 8, level = 0;
  while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
  return level;
}

void moveToBin(int b) {
  int from = head.read(), to = binToAngle(b);
  int dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
  head.write(to);
  delay(300);
}

int greedy(int s) {
  int ties[NA], n = 0;
  float m = max(Q[s][0], max(Q[s][1], Q[s][2]));
  for (int a = 0; a < NA; a++) if (Q[s][a] == m) ties[n++] = a;
  return ties[random(n)];
}

float trainReward(int level, int level2, bool success) {
  float r = success ? 5.0 : 0.0;
  if (mode == 'n' && level2 > level) r += 1.0;
  if (mode == 'p') r += GAMMA * level2 - level;
  return r;
}

void trainRun() {
  for (int s = 0; s < N; s++) for (int a = 0; a < NA; a++) Q[s][a] = 0;
  Serial.print(F("# mode ")); Serial.println(mode);
  Serial.println(F("episode,steps,back_and_forth,success"));
  int firstSuccess = 0;
  for (int ep = 0; ep < EPISODES; ep++) {
    float eps = max(0.05, 1.0 - ep / 20.0);
    angleBin = random(N);
    moveToBin(angleBin);
    int level = readLevel(), lastA = 1, flips = 0, t = 0;
    bool success = false;
    while (t < MAX_STEPS && !success) {
      int s = angleBin;
      int a = (random(1000) < eps * 1000) ? random(NA) : greedy(s);
      if ((a == 0 && lastA == 2) || (a == 2 && lastA == 0)) flips++;
      int s2 = constrain(s + (a - 1), 0, N - 1);
      if (s2 != s) moveToBin(s2);
      int level2 = readLevel();
      success = (level2 == 3);
      float r = trainReward(level, level2, success);
      float target = success ? r : r + GAMMA * max(Q[s2][0], max(Q[s2][1], Q[s2][2]));
      Q[s][a] += ALPHA * (target - Q[s][a]);
      angleBin = s2; level = level2; lastA = a; t++;
    }
    if (success && !firstSuccess) firstSuccess = ep + 1;
    Serial.print(ep + 1); Serial.print(','); Serial.print(t); Serial.print(',');
    Serial.print(flips); Serial.print(','); Serial.println(success ? 1 : 0);
  }
  Serial.print(F("# first success at episode ")); Serial.println(firstSuccess);
  Serial.println(F("# type s, n or p for another run"));
}

void setup() {
  Serial.begin(115200);
  randomSeed(analogRead(A5));
  head.attach(SERVO_PIN);
  head.write(90);
  delay(500);
  Serial.println(F("# type s (sparse), n (naive bonus) or p (potential-based) to train"));
}

void loop() {
  if (Serial.available()) {
    char c = Serial.read();
    if (c == 's' || c == 'n' || c == 'p') { mode = c; trainRun(); }
  }
}
Serial Monitor (115200) — shape of the output (your numbers will differ) · Text
# type s (sparse), n (naive bonus) or p (potential-based) to train
# mode n
episode,steps,back_and_forth,success
1,<steps>,<flips>,<0/1>
...
# first success at episode <k>
# type s, n or p for another run

Debugging

SymptomLikely causeFix
Training reward rises but the task never succeedsreward hackingWatch the robot (print a greedy trace); remove repeatable event bonuses
Robot dithers next to the goalbonus for getting closer without penalty for moving awayUse Φ(s′) − Φ(s) (potential-based)
Shaped run learns slower than sparseΦ points the wrong way (higher far from the goal)Check Φ increases as the task progresses
Naive mode succeeds sometimes, fails other timesbonus size close to the goal rewardCompare bonus scale with the true reward; prefer potential shaping
Python results differ from the tabledifferent seed or ε scheduleUse the same seeds; compare averages over 5+ seeds
UNO resets during trainingservo on UNO 5 VSeparate supply, grounds joined

Real world

Reward design in the real world

Several orange industrial robot arms stacking yellow crates of bread on pallets in a bakery
KUKA robots palletizing bread at a bakery in Germany. Industrial cells like this are programmed, not learnt — but if you train such a task with RL, a reward like “crates stacked” must also check *how* they are stacked, or the robot may find a sloppy shortcut.Photo: KUKA Roboter GmbH, Bachmann · Public domain · Source: Wikimedia Commons · resized, converted to WebP
  • Grasping research: the DeepMind brick-flipping example on Wikipedia came from a grasp reward measured from the wrong point on the brick — a one-line bug in the reward, not the learning.
  • Game agents: an OpenAI agent in the CoastRunners boat game learnt to loop through three targets for points instead of finishing the race.
  • Practice in labs: engineers often start with a sparse true reward plus a small step cost, add potential-based progress terms, and always watch videos of the learnt behaviour — not only the reward graph.
  • Safety: OpenAI researchers listed reward hacking among five “concrete problems in AI safety” (Amodei et al., 2016).
A tracked remotely operated robot with a long manipulator arm displayed on a red carpet at an exhibition
Daksh, a remotely controlled robot from India’s R&DE(E) for handling hazardous objects. On machines like this a wrong reward is not a funny bug — which is why such systems are tele-operated, and why learnt policies are trained and checked in simulation first.Photo: Jjamwal · CC BY 3.0 · Source: Wikimedia Commons · resized, converted to WebP

Think

Think about it

Think

You give a robot arm +1 for every centimetre the block moves towards the bin. What loophole might it find? How would you fix it with a potential function?

Experiment

Try it in the lab

About 70 minutes. Python 3 for the rail; the light-seeker hardware for the bench comparison.

Experiment

Break a reward, then fix it

You will need

  • `reward_shaping_lab.py` (Python 3, no libraries)
  • Reward shaping lab (in your browser)
  • Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, jumper wires
  • LED torch on a stand

Steps

  1. Run `reward_shaping_lab.py`. Copy the results table and both hack traces into your notebook.
  2. Change the “+2 per grasp” bonus to +0.5. Does the hack disappear? Is that a safe fix or just a lucky one?
  3. Write your own bad reward: “+1 for every step while holding the block”. Predict the hack, add it to the `kinds` list and run.
  4. Make Φ wrong on purpose (for example, flip its sign). What happens to the potential-based results — does it break the task or only slow learning?
  5. On the bench, run modes s, n and p twice each. Record first success and total back-and-forth moves.
  6. In the reward shaping lab, find the highest return you can get under Naive shaped without picking anything.

Interactive · Reward shaping lab · Educational

Browser toy only — NOT MuJoCo / Isaac / stable-baselines reward APIs. Demonstrates sparse vs dense vs naive shaping failure modes.

Only +1 on successful pick, 0 otherwise. Honest but hard to learn from few trials.

Phase

idle

Episode return

0.00

Hack risk (toy)

10%

toy pick cell · EE → object

Choose a reward mode, then step a toy pick episode.

A shaped reward makes learning faster only when it agrees with the true goal. Repeatable event bonuses and one-sided “closer” bonuses create loops the robot happily farms. Potential-based shaping gives the speed without changing what the best policy is.

Knowledge check

Check your understanding

Knowledge check

1. A sparse reward for pick-and-place is…

2. Reward hacking means…

3. With “+2 per grasp”, our robot learnt to…

4. Potential-based shaping has the form…

5. Why can a loop A → B → A not earn reward under potential shaping (γ = 1)?

6. The bicycle agent rode in tiny circles because…

7. Which result best shows that shaping helped without changing the goal?

8. The most reliable way to catch reward hacking is…

Mini assignment

Hands-on assignment

Mini assignment

Reward design sheet for a pick task

Design, break and fix a reward for a real manipulation task in your lab.

Deliverables

  • A reward design sheet for one manipulation task (e.g. “put the screwdriver in the tray”): true reward, step cost, and a potential function Φ with a short justification.
  • Two “bad” rewards you invented, with the hack you predicted and what the Python rail (or your own version) actually did.
  • Your results table for at least four training rewards, judged on the true task over 5 seeds.
  • The bench log for modes s, n and p (two runs each) with first success and back-and-forth counts.

Do this, step by step

  1. Pick one manipulation task and write its true reward, step cost and potential Φ.
  2. Invent two bad rewards and predict the hack each one invites.
  3. Run reward_shaping_lab.py and judge at least four training rewards on the true task over 5 seeds.
  4. Bench: upload shaping_compare.ino; type s, n and p in the Serial Monitor, two runs each.
  5. Record first success and back-and-forth counts for each mode.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. A high training reward with a low true success rate is the sign of a hack — always look at both columns.
  2. Potential-based shaping must use F = γΦ(s′) − Φ(s) with the same GAMMA (0.9) as the learner.
  3. Mode n gives +1 for brighter and nothing for darker, so swinging back and forth pays — count those moves.

Challenges (pick by level)

  • 🟢 Beginner: List three rewards a school could give students that can be “hacked” (Goodhart’s law: when a measure becomes a target, it stops being a good measure). Suggest a fix for each.
  • 🟡 Intermediate: Add a second block to the rail and design a Φ that guides the robot to place both. Check your Φ never rewards un-placing a block.
  • 🟠 Advanced: Try a subgoal-based Φ (e.g. +5 once holding, +10 once placed) as a potential, and compare it with the distance-based Φ.
  • 🔴 Expert: Prove on paper that for any path s₀ → … → s_T, the discounted sum of F = γΦ(s′) − Φ(s) equals γᵀΦ(s_T) − Φ(s₀).
  • ⚫ Research: Read the Reward hacking article on Wikipedia and pick one example. Write how potential-based shaping, a trip wire, or a better state would or would not have prevented it.

Next step

All our learners so far used a table: one number for every (state, action). That worked for 15 cells, 110 rail states and 9 angle bins. A real arm has thousands of angles and a camera image. In Lesson 6 — Why Tabular Q Breaks you will count how fast tables explode and see why robots need function approximation.

Look back: Lesson 4: Episodes, Exploration & Exploitation. Next: Lesson 6: Why Tabular Q Breaks. Related: Stage 13 — Perception, Planning & Manipulation.

Sources

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in