Advanced Robotics / Reinforcement Learning · Lesson 05
Reward Shaping for Manipulation
Est. 70–85 min
Why
Why this lesson matters
For manipulation, the true goal is usually sparse: “+10 when the block is in the bin”, nothing else. A robot exploring randomly may take hundreds of episodes to stumble on that reward even once. Reward shaping adds extra in-between rewards to guide it — but a careless bonus changes what the robot learns to want. This is called reward hacking or specification gaming.
Today you will train the same Q-learner on a pick-and-place rail with six different training rewards, judge every result on the *true* task, watch two robots hack their rewards, and learn the one form of shaping that is mathematically safe: potential-based shaping (Ng, Harada & Russell, 1999). On the bench, the light-seeker will compare sparse, naive and potential-based rewards live.
Concept
Sparse, dense and shaped rewards
1. Sparse vs dense rewards
| Type | Example for pick-and-place | Good | Bad |
|---|---|---|---|
| Sparse | +10 only when the block lands in the bin | says exactly what you want; hard to hack | very slow to learn — the robot rarely sees any reward |
| Step cost | sparse, plus −0.1 per step | encourages finishing quickly | still sparse at the start |
| Dense / shaped | sparse, plus bonuses for progress (getting closer, grasping) | fast learning signal | easy to create loopholes the robot exploits |

2. Reward hacking: the robot finds a shortcut
Reward hacking happens when an agent maximises the reward as written without achieving what the designer intended. Famous examples on Wikipedia: a bicycle-riding agent rewarded for moving towards the goal (but not punished for moving away) learnt to ride in tiny circles; a 2004 Lego Mindstorms robot rewarded for staying on a path learnt to zig-zag backwards over the straight part; a DeepMind arm flipped a brick because the grasp reward used the wrong reference point on the brick.

3. Potential-based shaping: the safe way
Ng, Harada and Russell proved in 1999 that if the extra reward has the form F(s, a, s′) = γ·Φ(s′) − Φ(s) for some “potential” function Φ over states, then the optimal policy does not change. And without further knowledge of the MDP, this is the *only* kind of shaping with that guarantee. Φ can be any sensible “how close am I to done” score — for example minus the distance to the next subgoal.
def potential(s):
gripper, block = s
if block == "H": # holding: get to the bin
return 10 - abs(BIN - gripper)
return -abs(block - gripper) - abs(BIN - block) # go to the block first
shaped_r = true_r + GAMMA * potential(s2) - potential(s)Visual
Visualise it: three reward modes
Switch between sparse, dense and naive shaped rewards on a toy pick episode — and try to “wiggle” your way to a high return without picking anything. (Educational simulation.)
Interactive · Reward shaping lab · Educational
Only +1 on successful pick, 0 otherwise. Honest but hard to learn from few trials.
Phase
idle
Episode return
0.00
Hack risk (toy)
10%
Choose a reward mode, then step a toy pick episode.
Code lab
Code lab: six training rewards, one true task
reward_shaping_lab.py puts a gripper on a 10-cell rail with the block at cell 2 and the bin at cell 8. Actions: LEFT, RIGHT, GRASP, RELEASE. The true reward is +10 for releasing the block into the bin. Tabular Q-learning (α 0.3, γ 0.95, ε 1 → 0.05) trains for 600 episodes with each of six training rewards and 5 seeds; every result is then measured with a greedy test on the true task.
"""reward_shaping_lab.py - Stage 14 L05: shaping the reward for a pick-and-place task.
A gripper slides along a 10-cell rail. A block starts at cell 2, the bin is at cell 8.
State : (gripper cell 0..9, block cell 0..9 or 'H' = held)
Actions: 0 LEFT, 1 RIGHT, 2 GRASP, 3 RELEASE
TRUE task reward (what we really want): +10 when the block is released into the bin,
episode ends. Nothing else. Max 60 steps.
The robot learns with tabular Q-learning (alpha 0.3, gamma 0.95, epsilon 1 -> 0.05)
under four different TRAINING rewards. We always judge it on the TRUE task.
Standard library only.
"""
import random
N, BIN, BLOCK0, MAX_STEPS, GAMMA = 10, 8, 2, 60, 0.95
LEFT, RIGHT, GRASP, RELEASE = range(4)
def step(s, a):
g, b = s
if a == LEFT:
g = max(0, g - 1)
elif a == RIGHT:
g = min(N - 1, g + 1)
elif a == GRASP and b == g:
b = "H"
elif a == RELEASE and b == "H":
if g == BIN:
return (g, BIN), True, True # placed in bin: success
b = g # dropped on the rail
return (g, b), False, False
def potential(s):
"""Phi(s): higher = closer to finishing. Distance-to-subgoal, plus a step for holding."""
g, b = s
if b == "H":
return 10 - abs(BIN - g) # carrying: go to the bin
return -abs(b - g) - abs(BIN - b) # not carrying: go to the block, then it must travel
def train_reward(kind, s, a, s2, success, grasped):
r = 10.0 if success else 0.0
if kind == "sparse":
return r
if kind == "step cost":
return r - 0.1
if kind == "+2 per grasp":
return r + (2.0 if grasped else 0.0) # careless bonus
if kind in ("+1 when closer", "+3 when closer"):
closer = potential(s2) > potential(s)
bonus = 1.0 if kind == "+1 when closer" else 3.0
return r + (bonus if closer else 0.0) # rewards progress, never punishes going back
if kind == "potential-based":
return r + GAMMA * potential(s2) - potential(s) # Ng, Harada & Russell 1999
raise ValueError(kind)
def choose(Q, s, eps, rng):
q = Q.setdefault(s, [0.0] * 4)
if rng.random() < eps:
return rng.randrange(4)
m = max(q)
return rng.choice([a for a in range(4) if q[a] == m])
def train(kind, episodes=600, seed=0):
rng, Q, first_success = random.Random(seed), {}, None
for ep in range(episodes):
s, eps = (rng.randrange(N), BLOCK0), max(0.05, 1 - ep / 300)
for _ in range(MAX_STEPS):
a = choose(Q, s, eps, rng)
s2, success, done = step(s, a)
grasped = s[1] != "H" and s2[1] == "H"
r = train_reward(kind, s, a, s2, success, grasped)
q2 = 0.0 if done else max(Q.setdefault(s2, [0.0] * 4))
Q[s][a] += 0.3 * (r + GAMMA * q2 - Q[s][a])
s = s2
if done:
if first_success is None:
first_success = ep + 1
break
return Q, first_success
def test(Q, episodes=200, seed=123):
"""Greedy test on the TRUE task: success rate, steps when successful, grasps per episode."""
rng = random.Random(seed)
wins, steps_won, grasps = 0, 0, 0
for _ in range(episodes):
s = (rng.randrange(N), BLOCK0)
for t in range(MAX_STEPS):
a = choose(Q, s, 0.0, rng)
s2, success, done = step(s, a)
grasps += s[1] != "H" and s2[1] == "H"
s = s2
if done:
wins += 1
steps_won += t + 1
break
return wins / episodes, (steps_won / wins if wins else float("nan")), grasps / episodes
def main():
kinds = ["sparse", "step cost", "+2 per grasp", "+1 when closer", "+3 when closer", "potential-based"]
print("Pick-and-place on a 10-cell rail (block at 2, bin at 8). 600 training episodes x 5 seeds.")
print("Every result below is measured on the TRUE task (+10 only for block in bin).\n")
print(f" {'training reward':17s} {'first success (episode)':>26s} {'greedy success':>15s} {'steps':>6s} {'grasps/ep':>10s}")
for kind in kinds:
firsts, succ, steps, gr = [], 0.0, [], 0.0
for seed in range(5):
Q, first = train(kind, seed=seed)
sr, st, g = test(Q, seed=200 + seed)
firsts.append(first)
succ += sr / 5
gr += g / 5
if sr > 0:
steps.append(st)
avg_steps = sum(steps) / len(steps) if steps else float("nan")
print(f" {kind:17s} {str(firsts):>26s} {succ * 100:13.0f} % {avg_steps:6.1f} {gr:10.2f}")
print("\nWhat does the '+2 per grasp' robot actually do? (greedy, seed 0, start at cell 0)")
Q, _ = train("+2 per grasp", seed=0)
s, names, trace = (0, BLOCK0), "LRGD", []
for _ in range(24):
a = choose(Q, s, 0.0, random.Random(1))
trace.append(names[a])
s, _, done = step(s, a)
if done:
trace.append("| DONE")
break
print(" " + " ".join(trace))
print(" (L/R = move, G = grasp, D = release/drop)")
print("\nWhat does the '+3 when closer' robot do? (greedy, seed 0, start at cell 0)")
Q, _ = train("+3 when closer", seed=0)
s, trace = (0, BLOCK0), []
for _ in range(24):
a = choose(Q, s, 0.0, random.Random(1))
trace.append(names[a])
s, _, done = step(s, a)
if done:
trace.append("| DONE")
break
print(" " + " ".join(trace))
print("\nWhy loops cannot be farmed with potential-based shaping (gamma = 1 for clarity):")
a, b = (2, BLOCK0), (3, BLOCK0) # gripper on the block, then one step away
closer = lambda x, y: 3.0 if potential(y) > potential(x) else 0.0
print(f" '+3 when closer' loop A->B->A earns {closer(a, b) + closer(b, a):+.1f} every 2 steps")
print(f" potential-based loop A->B->A earns {(potential(b) - potential(a)) + (potential(a) - potential(b)):+.1f}"
" (the bonuses telescope: only the start and end state matter)")
if __name__ == "__main__":
main()Output (run with python3 reward_shaping_lab.py)
Pick-and-place on a 10-cell rail (block at 2, bin at 8). 600 training episodes x 5 seeds. Every result below is measured on the TRUE task (+10 only for block in bin). training reward first success (episode) greedy success steps grasps/ep sparse [149, 72, 125, None, 46] 80 % 18.7 1.06 step cost [72, 41, 58, 55, 56] 100 % 11.1 1.00 +2 per grasp [None, 67, None, None, 44] 0 % nan 28.66 +1 when closer [139, 66, 101, 141, 63] 100 % 11.1 1.00 +3 when closer [139, 66, 101, 141, 63] 0 % nan 1.00 potential-based [47, 41, 40, 35, 33] 100 % 11.1 1.00 What does the '+2 per grasp' robot actually do? (greedy, seed 0, start at cell 0) R R G D G D G D G D G D G D G D G D G D G D G D (L/R = move, G = grasp, D = release/drop) What does the '+3 when closer' robot do? (greedy, seed 0, start at cell 0) R R G R R R R R R L R L R L R L R L R L R L R L Why loops cannot be farmed with potential-based shaping (gamma = 1 for clarity): '+3 when closer' loop A->B->A earns +3.0 every 2 steps potential-based loop A->B->A earns +0.0 (the bonuses telescope: only the start and end state matter)
Try it: choose Sparse, Dense or Naive shaped and step a toy pick episode. Under Naive shaped, try Wiggle and watch the return grow without any pick. (Educational simulation.)
Interactive · Reward shaping lab · Educational
Only +1 on successful pick, 0 otherwise. Honest but hard to learn from few trials.
Phase
idle
Episode return
0.00
Hack risk (toy)
10%
Choose a reward mode, then step a toy pick episode.
Bench build: compare rewards on the light-seeker
shaping_compare.ino is the Lesson 4 Q-learner with a choice of training reward, typed into the Serial Monitor: s = sparse (+5 at light level 3 only), n = naive (+1 whenever the light gets brighter, nothing for darker), p = potential-based (γ·level′ − level). Each run clears the Q-table and trains 40 episodes, logging steps, success and the number of back-and-forth moves.
| Part | Connection | Note |
|---|---|---|
| SG90/MG90-class servo | signal → D9 | the same light-seeker head |
| Servo supply | red → separate 5–6 V, brown → supply GND | join supply GND to UNO GND |
| LDR + 10 kΩ | LDR 5 V → A0, 10 kΩ A0 → GND | calibrate LEVEL_EDGES (Lesson 2) |
// shaping_compare.ino - Stage 14 L05
// Same Q-learning light-seeker as L04, but you choose the TRAINING reward from the Serial Monitor:
// 's' = sparse : +5 only when light level 3 is reached (episode terminates)
// 'n' = naive bonus : sparse + 1 whenever the light level goes UP (nothing for going down)
// 'p' = potential : sparse + (GAMMA * phi(s') - phi(s)), phi = light level (Ng et al. 1999)
// Each choice clears the Q-table and trains 40 episodes. The log counts "back-and-forth" moves
// (LEFT right after RIGHT or the reverse) - the tell-tale sign of a robot farming a bonus.
//
// Wiring: same as L01-L04. Servo signal D9 on its OWN 5-6 V supply (grounds joined),
// LDR 5V -> A0, 10k A0 -> GND. Serial 115200, line ending "No line ending" or "Newline".
#include <Servo.h>
const int SERVO_PIN = 9, LDR_PIN = A0;
const int N = 9, NA = 3, MAX_STEPS = 25, EPISODES = 40;
const int LEVEL_EDGES[3] = {300, 550, 800}; // calibrate for your room (see L02)
const float ALPHA = 0.3, GAMMA = 0.9;
Servo head;
float Q[N][NA];
char mode = 0;
int angleBin = 4;
int binToAngle(int b) { return 30 + 15 * b; }
int readLevel() {
long sum = 0;
for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
int raw = sum / 8, level = 0;
while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
return level;
}
void moveToBin(int b) {
int from = head.read(), to = binToAngle(b);
int dir = (to > from) ? 1 : -1;
for (int a = from; a != to; a += dir) { head.write(a); delay(8); }
head.write(to);
delay(300);
}
int greedy(int s) {
int ties[NA], n = 0;
float m = max(Q[s][0], max(Q[s][1], Q[s][2]));
for (int a = 0; a < NA; a++) if (Q[s][a] == m) ties[n++] = a;
return ties[random(n)];
}
float trainReward(int level, int level2, bool success) {
float r = success ? 5.0 : 0.0;
if (mode == 'n' && level2 > level) r += 1.0;
if (mode == 'p') r += GAMMA * level2 - level;
return r;
}
void trainRun() {
for (int s = 0; s < N; s++) for (int a = 0; a < NA; a++) Q[s][a] = 0;
Serial.print(F("# mode ")); Serial.println(mode);
Serial.println(F("episode,steps,back_and_forth,success"));
int firstSuccess = 0;
for (int ep = 0; ep < EPISODES; ep++) {
float eps = max(0.05, 1.0 - ep / 20.0);
angleBin = random(N);
moveToBin(angleBin);
int level = readLevel(), lastA = 1, flips = 0, t = 0;
bool success = false;
while (t < MAX_STEPS && !success) {
int s = angleBin;
int a = (random(1000) < eps * 1000) ? random(NA) : greedy(s);
if ((a == 0 && lastA == 2) || (a == 2 && lastA == 0)) flips++;
int s2 = constrain(s + (a - 1), 0, N - 1);
if (s2 != s) moveToBin(s2);
int level2 = readLevel();
success = (level2 == 3);
float r = trainReward(level, level2, success);
float target = success ? r : r + GAMMA * max(Q[s2][0], max(Q[s2][1], Q[s2][2]));
Q[s][a] += ALPHA * (target - Q[s][a]);
angleBin = s2; level = level2; lastA = a; t++;
}
if (success && !firstSuccess) firstSuccess = ep + 1;
Serial.print(ep + 1); Serial.print(','); Serial.print(t); Serial.print(',');
Serial.print(flips); Serial.print(','); Serial.println(success ? 1 : 0);
}
Serial.print(F("# first success at episode ")); Serial.println(firstSuccess);
Serial.println(F("# type s, n or p for another run"));
}
void setup() {
Serial.begin(115200);
randomSeed(analogRead(A5));
head.attach(SERVO_PIN);
head.write(90);
delay(500);
Serial.println(F("# type s (sparse), n (naive bonus) or p (potential-based) to train"));
}
void loop() {
if (Serial.available()) {
char c = Serial.read();
if (c == 's' || c == 'n' || c == 'p') { mode = c; trainRun(); }
}
}# type s (sparse), n (naive bonus) or p (potential-based) to train
# mode n
episode,steps,back_and_forth,success
1,<steps>,<flips>,<0/1>
...
# first success at episode <k>
# type s, n or p for another runDebugging
| Symptom | Likely cause | Fix |
|---|---|---|
| Training reward rises but the task never succeeds | reward hacking | Watch the robot (print a greedy trace); remove repeatable event bonuses |
| Robot dithers next to the goal | bonus for getting closer without penalty for moving away | Use Φ(s′) − Φ(s) (potential-based) |
| Shaped run learns slower than sparse | Φ points the wrong way (higher far from the goal) | Check Φ increases as the task progresses |
| Naive mode succeeds sometimes, fails other times | bonus size close to the goal reward | Compare bonus scale with the true reward; prefer potential shaping |
| Python results differ from the table | different seed or ε schedule | Use the same seeds; compare averages over 5+ seeds |
| UNO resets during training | servo on UNO 5 V | Separate supply, grounds joined |
Real world
Reward design in the real world

- Grasping research: the DeepMind brick-flipping example on Wikipedia came from a grasp reward measured from the wrong point on the brick — a one-line bug in the reward, not the learning.
- Game agents: an OpenAI agent in the CoastRunners boat game learnt to loop through three targets for points instead of finishing the race.
- Practice in labs: engineers often start with a sparse true reward plus a small step cost, add potential-based progress terms, and always watch videos of the learnt behaviour — not only the reward graph.
- Safety: OpenAI researchers listed reward hacking among five “concrete problems in AI safety” (Amodei et al., 2016).

Think
Think about it
Think
You give a robot arm +1 for every centimetre the block moves towards the bin. What loophole might it find? How would you fix it with a potential function?
Experiment
Try it in the lab
About 70 minutes. Python 3 for the rail; the light-seeker hardware for the bench comparison.
Experiment
Break a reward, then fix it
You will need
- `reward_shaping_lab.py` (Python 3, no libraries)
- Reward shaping lab (in your browser)
- Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, jumper wires
- LED torch on a stand
Steps
- Run `reward_shaping_lab.py`. Copy the results table and both hack traces into your notebook.
- Change the “+2 per grasp” bonus to +0.5. Does the hack disappear? Is that a safe fix or just a lucky one?
- Write your own bad reward: “+1 for every step while holding the block”. Predict the hack, add it to the `kinds` list and run.
- Make Φ wrong on purpose (for example, flip its sign). What happens to the potential-based results — does it break the task or only slow learning?
- On the bench, run modes s, n and p twice each. Record first success and total back-and-forth moves.
- In the reward shaping lab, find the highest return you can get under Naive shaped without picking anything.
Interactive · Reward shaping lab · Educational
Only +1 on successful pick, 0 otherwise. Honest but hard to learn from few trials.
Phase
idle
Episode return
0.00
Hack risk (toy)
10%
Choose a reward mode, then step a toy pick episode.
A shaped reward makes learning faster only when it agrees with the true goal. Repeatable event bonuses and one-sided “closer” bonuses create loops the robot happily farms. Potential-based shaping gives the speed without changing what the best policy is.
Knowledge check
Check your understanding
Knowledge check
1. A sparse reward for pick-and-place is…
2. Reward hacking means…
3. With “+2 per grasp”, our robot learnt to…
4. Potential-based shaping has the form…
5. Why can a loop A → B → A not earn reward under potential shaping (γ = 1)?
6. The bicycle agent rode in tiny circles because…
7. Which result best shows that shaping helped without changing the goal?
8. The most reliable way to catch reward hacking is…
Mini assignment
Hands-on assignment
Mini assignment
Reward design sheet for a pick task
Design, break and fix a reward for a real manipulation task in your lab.
Deliverables
- A reward design sheet for one manipulation task (e.g. “put the screwdriver in the tray”): true reward, step cost, and a potential function Φ with a short justification.
- Two “bad” rewards you invented, with the hack you predicted and what the Python rail (or your own version) actually did.
- Your results table for at least four training rewards, judged on the true task over 5 seeds.
- The bench log for modes s, n and p (two runs each) with first success and back-and-forth counts.
Do this, step by step
- Pick one manipulation task and write its true reward, step cost and potential Φ.
- Invent two bad rewards and predict the hack each one invites.
- Run
reward_shaping_lab.pyand judge at least four training rewards on the true task over 5 seeds. - Bench: upload
shaping_compare.ino; type s, n and p in the Serial Monitor, two runs each. - Record first success and back-and-forth counts for each mode.
Stuck? Hints, one at a time
- A high training reward with a low true success rate is the sign of a hack — always look at both columns.
- Potential-based shaping must use F = γΦ(s′) − Φ(s) with the same
GAMMA(0.9) as the learner. - Mode n gives +1 for brighter and nothing for darker, so swinging back and forth pays — count those moves.
Challenges (pick by level)
- 🟢 Beginner: List three rewards a school could give students that can be “hacked” (Goodhart’s law: when a measure becomes a target, it stops being a good measure). Suggest a fix for each.
- 🟡 Intermediate: Add a second block to the rail and design a Φ that guides the robot to place both. Check your Φ never rewards un-placing a block.
- 🟠 Advanced: Try a subgoal-based Φ (e.g. +5 once holding, +10 once placed) as a potential, and compare it with the distance-based Φ.
- 🔴 Expert: Prove on paper that for any path s₀ → … → s_T, the discounted sum of F = γΦ(s′) − Φ(s) equals γᵀΦ(s_T) − Φ(s₀).
- ⚫ Research: Read the Reward hacking article on Wikipedia and pick one example. Write how potential-based shaping, a trip wire, or a better state would or would not have prevented it.
Next step
All our learners so far used a table: one number for every (state, action). That worked for 15 cells, 110 rail states and 9 angle bins. A real arm has thousands of angles and a camera image. In Lesson 6 — Why Tabular Q Breaks you will count how fast tables explode and see why robots need function approximation.
Look back: Lesson 4: Episodes, Exploration & Exploitation. Next: Lesson 6: Why Tabular Q Breaks. Related: Stage 13 — Perception, Planning & Manipulation.
Sources
- Ng, Harada & Russell (1999) — Policy invariance under reward transformations (PDF, UC Berkeley) — potential-based shaping F = γΦ(s′) − Φ(s) preserves the optimal policy and is necessary without further assumptions; the bicycle “tiny circle” bug; distance- and subgoal-based potentials.
- ML Anthology — Policy Invariance Under Reward Transformations (ICML 1999) — abstract and citation.
- Wikipedia — Reward hacking — definition (specification gaming), Goodhart’s law, Amodei et al. 2016 concrete problems, 2004 Lego Mindstorms path robot, DeepMind brick-flipping grasp reward, OpenAI CoastRunners.
- Gymnasium — Create a Custom Environment — sparse-reward pitfall, step penalty and distance-based reward options.
- Wikipedia — Q-learning — the learner used in this lesson.
- Arduino Servo library (GitHub) — controlling RC hobby servos.
- Photos: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).