Advanced Robotics / Reinforcement Learning · Lesson 02
States, Actions & Rewards
Est. 60–75 min
Why
Why this lesson matters
Before a robot can learn anything with reinforcement learning, a human must design the problem: what the robot is allowed to see (the state), what it is allowed to do (the actions), what counts as good (the reward), and when an attempt is over (the episode). This design is called a Markov decision process (MDP). A badly designed MDP makes learning slow or impossible, no matter how clever the algorithm.
In this lesson you will build a small desk-pick task as a Python environment with the same reset() / step() shape used by Gymnasium, the standard RL toolkit. You will see why leaving out one sensor changes the result, what γ (the discount) does to a return, and how terminated differs from truncated. On the bench, the UNO will log its own (s, a, r, s′) lines as CSV.
Concept
The Markov decision process, piece by piece
1. The four parts of an MDP
| Symbol | Name | Meaning | In our desk-pick task |
|---|---|---|---|
| S | State space | every situation the robot can be in | (gripper cell 0–4, jaw open/closed, holding yes/no) — 20 states |
| A | Action space | every choice the robot can make | LEFT, RIGHT, CLOSE, OPEN, LIFT — 5 actions |
| P(s′ | s, a) | Transition | what happens next, possibly random | CLOSE over the block grabs it 80 % of the time |
| R | Reward | a number after each step | −1 per step, −3 lifting empty, +10 lifting the block |

2. The Markov property: the present is enough
The “Markov” in MDP means: the next state depends only on the current state and action, not on the full history. If you know where the gripper is, whether the jaw is closed and whether it is holding the block, you know everything needed to predict what LIFT will do. The name comes from the Russian mathematician Andrey Markov, who studied chains of random events with this property.

3. Reward, return and discount
A reward is one number for one step. The return G is the total for the whole episode. Usually future rewards are multiplied by a discount factor γ (gamma) between 0 and 1: G = r₁ + γ·r₂ + γ²·r₃ + … With γ = 1 every step counts equally. With γ = 0.5 the robot is short-sighted: a reward 5 steps away is worth only 1/32 of its face value. In our run, the same successful episode has return +6.00 at γ = 1, +3.12 at γ = 0.9 and −1.25 at γ = 0.5 — the +10 at the end is discounted so much that the step costs win.

4. Episodes: terminated vs truncated
An episode is one attempt from reset() until it ends. Gymnasium’s step() returns two separate flags. terminated means the task itself ended — success (block lifted) or a real failure (fell into lava, robot crashed). truncated means we stopped for an outside reason, usually a time limit. Learning algorithms treat them differently: after “terminated” there is no future reward; after “truncated” the future simply was not observed. Older Gym code had one done flag; Gymnasium replaced it with these two because mixing them up confuses learning.
obs, info = env.reset(seed=42)
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()Visual
Visualise it: trace one transition
Pick a state and action in a toy transition table and read the reward and next state. This is exactly what env.step() does. (Educational simulation.)
Interactive · RL MDP explorer · Educational
State s
Approach
Last reward r
—
Episode return
0.00 · 0 steps
Actions a (from current state)
Pick a state, then take an action. Watch reward + next state.
Toy transition table (partial)
| s | a | r | s′ |
|---|---|---|---|
| Approach | Move closer | -0.10 | At grasp |
| Approach | Close gripper | -1.00 | Fail |
| Approach | Raise | -0.20 | Approach |
| Approach | Retreat | -0.05 | Approach |
| At grasp | Move closer | -0.05 | At grasp |
| At grasp | Close gripper | 0.50 | Lift |
| At grasp | Raise | -1.00 | Fail |
| At grasp | Retreat | -0.10 | Approach |
| Lift | Move closer | -0.05 | Lift |
| Lift | Close gripper | -0.02 | Lift |
| Lift | Raise | 1.00 | Done |
| Lift | Retreat | -0.50 | Fail |
| Done | Move closer | 0.00 | Done |
| Done | Close gripper | 0.00 | Done |
| Done | Raise | 0.00 | Done |
| Done | Retreat | 0.00 | Done |
Code lab
Code lab: build a desk-pick environment
desk_pick_env.py writes the desk-pick task as a class with reset(seed) and step(action) returning (obs, reward, terminated, truncated, info) — the same shape as gymnasium.Env, but with only the Python standard library so you can run it on any laptop. It then tests three hand-written policies (no learning yet) on the same 500 seeds.
"""desk_pick_env.py - Stage 14 L02: a robot task written as an MDP.
A Gymnasium-style environment (same reset/step shape as gymnasium.Env),
written with the standard library only so it runs anywhere.
The desk is a line of 5 cells. The block sits in cell 3.
State : (gripper cell 0..4, jaw "open"/"closed", holding True/False)
Actions: 0 LEFT, 1 RIGHT, 2 CLOSE, 3 OPEN, 4 LIFT
Reward : -1 every step (time costs), -3 for lifting an empty gripper,
+10 for lifting the block (success, episode terminated).
Physics: closing over the block grabs it only 80 % of the time (slip).
Episode is truncated after 20 steps (a time limit, not a task outcome).
"""
import random
LEFT, RIGHT, CLOSE, OPEN, LIFT = range(5)
NAMES = ["LEFT", "RIGHT", "CLOSE", "OPEN", "LIFT"]
class DeskPickEnv:
def __init__(self, block_cell=3, grip_success=0.8, max_steps=20, show_holding=True):
self.block_cell = block_cell
self.grip_success = grip_success
self.max_steps = max_steps
self.show_holding = show_holding # False = the sensor that tells "holding" is missing
self.rng = random.Random()
def _obs(self):
holding = self.holding if self.show_holding else None
return (self.cell, "closed" if self.closed else "open", holding)
def reset(self, seed=None):
if seed is not None:
self.rng.seed(seed)
self.cell, self.closed, self.holding, self.t = 0, False, False, 0
return self._obs(), {"block_cell": self.block_cell}
def step(self, action):
assert action in range(5), "action outside the action space"
self.t += 1
reward, terminated = -1.0, False
if action == LEFT and not self.closed:
self.cell = max(0, self.cell - 1)
elif action == RIGHT and not self.closed:
self.cell = min(4, self.cell + 1)
elif action == CLOSE and not self.closed:
self.closed = True
self.holding = (self.cell == self.block_cell and self.rng.random() < self.grip_success)
elif action == OPEN:
self.closed, self.holding = False, False
elif action == LIFT:
if self.holding:
reward, terminated = 10.0, True
else:
reward = -3.0
self.closed = False # arm comes back up and re-opens
truncated = (not terminated) and self.t >= self.max_steps
return self._obs(), reward, terminated, truncated, {"t": self.t}
# ---- three policies: rules that map an observation to an action ----
def random_policy(obs, rng):
return rng.randrange(5)
def careful_policy(obs, rng):
"""Uses the 'holding' sensor: re-grip if the first grip slipped."""
cell, jaw, holding = obs
if cell < 3:
return RIGHT if jaw == "open" else OPEN
if jaw == "open":
return CLOSE
return LIFT if holding else OPEN
def blind_policy(obs, rng):
"""No holding sensor: always close then lift and hope."""
cell, jaw, _ = obs
if cell < 3:
return RIGHT if jaw == "open" else OPEN
return CLOSE if jaw == "open" else LIFT
def run_episode(env, policy, seed, gamma=1.0, trace=False):
obs, info = env.reset(seed=seed)
prng = random.Random(seed + 1000)
g, discount, steps = 0.0, 1.0, 0
while True:
a = policy(obs, prng)
nobs, r, term, trunc, info = env.step(a)
if trace:
print(f" t={info['t']:2d} s={obs} a={NAMES[a]:5s} r={r:+5.1f} s'={nobs}"
+ (" TERMINATED" if term else "") + (" TRUNCATED" if trunc else ""))
g += discount * r
discount *= gamma
steps += 1
obs = nobs
if term or trunc:
return g, term, steps
def main():
env = DeskPickEnv()
print("1) One episode with the careful policy (seed 4) - every row is (s, a, r, s')")
g, term, steps = run_episode(env, careful_policy, seed=4, trace=True)
print(f" return = {g:+.1f} in {steps} steps")
print("\n1b) Same policy, seed 0 - the first grip slips, the holding sensor says False")
g, term, steps = run_episode(env, careful_policy, seed=0, trace=True)
print(f" return = {g:+.1f} in {steps} steps")
print("\n2) 500 episodes per policy (same seeds for a fair test)")
for name, pol, show in [("random", random_policy, True),
("careful (sees holding)", careful_policy, True),
("blind (no holding sensor)", blind_policy, False)]:
env = DeskPickEnv(show_holding=show)
res = [run_episode(env, pol, seed=s) for s in range(500)]
avg = sum(r[0] for r in res) / len(res)
succ = sum(1 for r in res if r[1])
print(f" {name:27s} avg return {avg:+6.2f} success {succ:3d}/500 avg steps {sum(r[2] for r in res)/500:5.1f}")
print("\n3) Same careful episode, different discount gamma")
for gamma in (1.0, 0.9, 0.5):
g, _, _ = run_episode(DeskPickEnv(), careful_policy, seed=4, gamma=gamma)
print(f" gamma {gamma:.1f}: discounted return {g:+.2f}")
print("\n4) Terminated vs truncated (random policy, 500 episodes)")
env = DeskPickEnv()
res = [run_episode(env, random_policy, seed=s) for s in range(500)]
print(f" terminated (task really ended) : {sum(1 for r in res if r[1])}")
print(f" truncated (20-step time limit) : {sum(1 for r in res if not r[1])}")
print("\n5) Size of this MDP")
states = 5 * 2 * 2
print(f" states {states} (5 cells x jaw x holding), actions 5, table entries {states * 5}")
if __name__ == "__main__":
main()Output (run with python3 desk_pick_env.py)
1) One episode with the careful policy (seed 4) - every row is (s, a, r, s') t= 1 s=(0, 'open', False) a=RIGHT r= -1.0 s'=(1, 'open', False) t= 2 s=(1, 'open', False) a=RIGHT r= -1.0 s'=(2, 'open', False) t= 3 s=(2, 'open', False) a=RIGHT r= -1.0 s'=(3, 'open', False) t= 4 s=(3, 'open', False) a=CLOSE r= -1.0 s'=(3, 'closed', True) t= 5 s=(3, 'closed', True) a=LIFT r=+10.0 s'=(3, 'closed', True) TERMINATED return = +6.0 in 5 steps 1b) Same policy, seed 0 - the first grip slips, the holding sensor says False t= 1 s=(0, 'open', False) a=RIGHT r= -1.0 s'=(1, 'open', False) t= 2 s=(1, 'open', False) a=RIGHT r= -1.0 s'=(2, 'open', False) t= 3 s=(2, 'open', False) a=RIGHT r= -1.0 s'=(3, 'open', False) t= 4 s=(3, 'open', False) a=CLOSE r= -1.0 s'=(3, 'closed', False) t= 5 s=(3, 'closed', False) a=OPEN r= -1.0 s'=(3, 'open', False) t= 6 s=(3, 'open', False) a=CLOSE r= -1.0 s'=(3, 'closed', True) t= 7 s=(3, 'closed', True) a=LIFT r=+10.0 s'=(3, 'closed', True) TERMINATED return = +4.0 in 7 steps 2) 500 episodes per policy (same seeds for a fair test) random avg return -26.95 success 22/500 avg steps 19.8 careful (sees holding) avg return +5.51 success 500/500 avg steps 5.5 blind (no holding sensor) avg return +5.02 success 500/500 avg steps 5.5 3) Same careful episode, different discount gamma gamma 1.0: discounted return +6.00 gamma 0.9: discounted return +3.12 gamma 0.5: discounted return -1.25 4) Terminated vs truncated (random policy, 500 episodes) terminated (task really ended) : 22 truncated (20-step time limit) : 478 5) Size of this MDP states 20 (5 cells x jaw x holding), actions 5, table entries 100
Try it: pick a state and an action, then read the reward and next state from the toy transition table. Write down three (s, a, r, s′) rows by hand. (Educational simulation — not MuJoCo, Isaac or a real RL library.)
Interactive · RL MDP explorer · Educational
State s
Approach
Last reward r
—
Episode return
0.00 · 0 steps
Actions a (from current state)
Pick a state, then take an action. Watch reward + next state.
Toy transition table (partial)
| s | a | r | s′ |
|---|---|---|---|
| Approach | Move closer | -0.10 | At grasp |
| Approach | Close gripper | -1.00 | Fail |
| Approach | Raise | -0.20 | Approach |
| Approach | Retreat | -0.05 | Approach |
| At grasp | Move closer | -0.05 | At grasp |
| At grasp | Close gripper | 0.50 | Lift |
| At grasp | Raise | -1.00 | Fail |
| At grasp | Retreat | -0.10 | Approach |
| Lift | Move closer | -0.05 | Lift |
| Lift | Close gripper | -0.02 | Lift |
| Lift | Raise | 1.00 | Done |
| Lift | Retreat | -0.50 | Fail |
| Done | Move closer | 0.00 | Done |
| Done | Close gripper | 0.00 | Done |
| Done | Raise | 0.00 | Done |
| Done | Retreat | 0.00 | Done |
Bench build: the UNO writes its own transition log
transition_logger.ino reuses the Lesson 1 light-seeking head but now speaks MDP. State = (angle bin 0–8, light level 0–3). Actions = LEFT, STAY, RIGHT by 15°. Reward = +1 if the light level went up, −1 if down, −0.1 otherwise. The episode terminates at light level 3 and is truncated after 30 steps. The actions are random on purpose — today we only collect data. Lesson 3 will learn from it.
| Part | Connection | Note |
|---|---|---|
| SG90/MG90-class servo | signal → D9 | LDR taped to the horn, shaded by a paper tube |
| Servo supply | red → separate 5–6 V, brown → supply GND | join supply GND to UNO GND |
| LDR + 10 kΩ | LDR 5 V → A0, 10 kΩ A0 → GND | measure dark/bright readings, then edit LEVEL_EDGES |
| Push button | D2 → button → GND | INPUT_PULLUP; press to start each episode (env.reset) |
// transition_logger.ino - Stage 14 L02
// The L01 light-seeker head, now written as an MDP and logged as (s, a, r, s').
// State s = (angle bin 0..8, light level 0..3)
// Action a = 0 LEFT (-15 deg), 1 STAY, 2 RIGHT (+15 deg) - chosen at random here
// Reward r = +1 if the light level went up, -1 if it went down, -0.1 otherwise (time cost)
// Episode: TERMINATED when light level 3 is reached, TRUNCATED after 30 steps.
//
// Wiring: same as L01. Servo signal D9 (servo on its OWN 5-6 V supply, grounds joined),
// LDR from 5V to A0 + 10k from A0 to GND, button D2 -> GND starts a new episode.
// Serial 115200 prints CSV you can paste into a spreadsheet or Python.
#include <Servo.h>
const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int MAX_STEPS = 30;
// Light thresholds: measure YOUR room first (see lesson) and edit these.
const int LEVEL_EDGES[3] = {300, 550, 800};
Servo head;
int angleBin = 4; // bin 4 = 90 degrees
int stepCount = 0;
int episode = 0;
bool running = false;
int binToAngle(int b) { return 30 + 15 * b; } // bins 0..8 -> 30..150 deg
int readLevel() {
long sum = 0;
for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
int raw = sum / 8;
int level = 0;
while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
return level;
}
void moveToBin(int b) {
int from = head.read(), to = binToAngle(b);
int dir = (to > from) ? 1 : -1;
for (int a = from; a != to; a += dir) { head.write(a); delay(10); }
head.write(to);
delay(350); // settle before reading the sensor
}
void startEpisode() {
episode++;
stepCount = 0;
angleBin = random(9); // random start state, like env.reset()
moveToBin(angleBin);
running = true;
Serial.print(F("# episode ")); Serial.println(episode);
}
void setup() {
Serial.begin(115200);
pinMode(BUTTON_PIN, INPUT_PULLUP);
randomSeed(analogRead(A5));
head.attach(SERVO_PIN);
head.write(90);
delay(500);
Serial.println(F("episode,t,s_bin,s_light,a,r,s2_bin,s2_light,terminated,truncated"));
Serial.println(F("# press the button to start an episode"));
}
void loop() {
if (!running) {
if (digitalRead(BUTTON_PIN) == LOW) { delay(30); startEpisode(); }
return;
}
int sBin = angleBin, sLight = readLevel();
int a = random(3); // 0 LEFT, 1 STAY, 2 RIGHT
int nextBin = constrain(sBin + (a - 1), 0, 8); // walls: cannot go past 30 or 150 deg
if (nextBin != sBin) moveToBin(nextBin);
angleBin = nextBin;
int s2Light = readLevel();
stepCount++;
float r = -0.1;
if (s2Light > sLight) r = 1.0;
else if (s2Light < sLight) r = -1.0;
bool terminated = (s2Light == 3);
bool truncated = !terminated && stepCount >= MAX_STEPS;
Serial.print(episode); Serial.print(',');
Serial.print(stepCount); Serial.print(',');
Serial.print(sBin); Serial.print(','); Serial.print(sLight); Serial.print(',');
Serial.print(a); Serial.print(','); Serial.print(r, 1); Serial.print(',');
Serial.print(angleBin); Serial.print(','); Serial.print(s2Light); Serial.print(',');
Serial.print(terminated ? 1 : 0); Serial.print(','); Serial.println(truncated ? 1 : 0);
if (terminated || truncated) {
running = false;
Serial.println(terminated ? F("# TERMINATED: reached level 3") : F("# TRUNCATED: 30-step limit"));
Serial.println(F("# press the button for the next episode"));
}
delay(150);
}episode,t,s_bin,s_light,a,r,s2_bin,s2_light,terminated,truncated
# press the button to start an episode
# episode 1
1,1,<bin>,<level>,<0-2>,<-0.1 / 1.0 / -1.0>,<bin>,<level>,0,0
...
# TERMINATED: reached level 3 (or) # TRUNCATED: 30-step limitDebugging
| Symptom | Likely cause | Fix |
|---|---|---|
| Every episode is TRUNCATED | level 3 threshold too high for your torch | Measure the bright reading and lower LEVEL_EDGES[2] |
| Every episode TERMINATES at step 1 | room already brighter than level 3 | Raise the thresholds or dim the room |
| Same (s, a) gives different s′ often | servo still shaking during reading, or flickering tube light | Increase the 350 ms settle delay; use a steady LED torch |
| Python: step() after the episode ended gives strange results | forgot to call reset() | Always reset when terminated or truncated is True |
| Python: return looks wrong | gamma multiplied in the wrong order | Multiply the reward by the current discount, then update discount *= gamma |
| UNO resets when the servo moves | servo on UNO 5 V | Separate supply, grounds joined |
Real world
MDPs in the real world

- Pole balancing (CartPole): a classic MDP example. The state describes where the cart and pole are and how fast they move; the actions push the cart left or right; the episode ends when the pole falls.
- Robot arms: state is often joint angles and speeds plus the object’s pose from the camera (Stage 13); actions are small joint or gripper commands.
- Mobile robots: state = pose and nearby obstacles (from an occupancy map); actions = velocity commands; reward = progress minus collisions.
- Design is the hard part: the Gymnasium guide itself warns that a poorly designed environment makes learning difficult or impossible no matter how good the algorithm is.

Think
Think about it
Think
You design a state for the light-seeker as just “light level 0–3”, without the angle. Is that a good state? What goes wrong?
Experiment
Try it in the lab
About 60 minutes. Python 3 for the desk-pick environment; the Lesson 1 light-seeker hardware for the logger.
Experiment
Design, run and log an MDP
You will need
- `desk_pick_env.py` (Python 3, no libraries)
- Toy MDP explorer (in your browser)
- Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
- Steady LED torch on a stand, spreadsheet software
Steps
- Run `desk_pick_env.py`. Copy the seed-0 trace into your notebook as a table with columns s, a, r, s′.
- Change `grip_success` from 0.8 to 0.5. Run again. How much bigger does the gap between the careful and blind policies get?
- Change the empty-lift penalty from −3 to −20. Does any policy’s average return change much? Why?
- On the bench, calibrate LEVEL_EDGES for your room, then run 5 episodes of the logger and save the CSV.
- In the spreadsheet, count how many episodes terminated vs were truncated. Find one (s, a) pair that gave two different s′.
- In the MDP explorer, find the action that ends an episode with the worst reward and the shortest path to the best reward.
Interactive · RL MDP explorer · Educational
State s
Approach
Last reward r
—
Episode return
0.00 · 0 steps
Actions a (from current state)
Pick a state, then take an action. Watch reward + next state.
Toy transition table (partial)
| s | a | r | s′ |
|---|---|---|---|
| Approach | Move closer | -0.10 | At grasp |
| Approach | Close gripper | -1.00 | Fail |
| Approach | Raise | -0.20 | Approach |
| Approach | Retreat | -0.05 | Approach |
| At grasp | Move closer | -0.05 | At grasp |
| At grasp | Close gripper | 0.50 | Lift |
| At grasp | Raise | -1.00 | Fail |
| At grasp | Retreat | -0.10 | Approach |
| Lift | Move closer | -0.05 | Lift |
| Lift | Close gripper | -0.02 | Lift |
| Lift | Raise | 1.00 | Done |
| Lift | Retreat | -0.50 | Fail |
| Done | Move closer | 0.00 | Done |
| Done | Close gripper | 0.00 | Done |
| Done | Raise | 0.00 | Done |
| Done | Retreat | 0.00 | Done |
An MDP is a design: choosing the state, actions, reward and episode rules decides what the robot can possibly learn. Missing information (partial observability) shows up as wasted actions and inconsistent transitions in the log.
Knowledge check
Check your understanding
Knowledge check
1. An MDP is usually written as the tuple…
2. The Markov property says the next state depends on…
3. With rewards −1, −1, +10 and γ = 0.9, the return is…
4. In Gymnasium, `truncated = True` usually means…
5. The blind policy without the holding sensor had a lower return because…
6. What must you call after `terminated` or `truncated` is True?
7. A very small γ (like 0.5) makes the agent…
8. In the logger, which part is NOT a design choice made by you?
Mini assignment
Hands-on assignment
Mini assignment
MDP card for your own robot task
Write a real robot task as a clean MDP and test it.
Deliverables
- An MDP card for a desk task of your choice: S, A, P (what is random?), R, γ, and the terminated / truncated rules.
- Your hand-made (s, a, r, s′) table for the seed-0 desk-pick episode, with the return computed at γ = 1 and γ = 0.9.
- The Python result with grip_success 0.5: careful vs blind average return, and your explanation.
- The UNO CSV for 5 episodes, with terminated/truncated counts and one example of an inconsistent transition.
Do this, step by step
- Write your MDP card: S, A, P, R, γ, and when an episode is terminated or truncated.
- Run
desk_pick_env.pyand fill the (s, a, r, s′) table for the seed-0 episode by hand. - Change
grip_successfrom 0.8 to 0.5 and compare the careful and blind policies. - Bench: upload
transition_logger.inoand save 5 episodes of CSV from the Serial Monitor. - Count terminated vs truncated (30 steps,
MAX_STEPS) and find one transition that looks inconsistent.
Stuck? Hints, one at a time
- Return at γ = 0.9: multiply the reward of step k by 0.9^k before adding — the first step uses 0.9^0 = 1.
- If every episode is truncated, your room never reaches light level 3 — lower
LEVEL_EDGES{300, 550, 800} to match your LDR readings. - An “inconsistent” transition is the same (s, a) giving a different s′ — that is the randomness P describes, not a bug.
Challenges (pick by level)
- 🟢 Beginner: Write Snakes and Ladders as an MDP. What is the state? What are the actions (hint: there are none — so what is it?). Now add a rule “you may choose to roll one dice or two”. What changes?
- 🟡 Intermediate: Add a second block in cell 1 to
desk_pick_env.py. Change the state so the policy can know which block it holds. How many states now? - 🟠 Advanced: Add
render()to the environment that prints the desk as text, like. . . [B] .with the gripper marked, like the simple text render in the Gymnasium custom-environment tutorial. - 🔴 Expert: Install Gymnasium (
pip install gymnasium), turnDeskPickEnvinto a realgymnasium.Envsubclass withDiscreteaction and observation spaces, and rungymnasium.utils.env_checker.check_envon it. - ⚫ Research: Read the Wikipedia article on Markov decision processes and find the name of the version where the state is not fully visible. Describe one robot task in your lab that is really of that kind.
Next step
Our policies today were written by hand. The next question is: how good is each state, and how should the robot choose actions to get the highest return? In Lesson 3 — Policies & Value you will compute value tables and read a greedy policy from them.
Look back: Lesson 1: Why Robots Learn. Next: Lesson 3: Policies & Value. Related: Stage 13 — Perception, Planning & Manipulation.
Sources
- Gymnasium — Env API (Farama Foundation) —
reset(seed)returns (observation, info);step(action)returns (observation, reward, terminated, truncated, info); terminated vs truncated and the removal ofdonein v0.26. - Gymnasium — Create a Custom Environment — environment design questions, GridWorld example,
Discrete(4)actions, reward and state-representation pitfalls,check_env. - Wikipedia — Markov decision process — the 4-tuple (S, A, Pₐ, Rₐ), discount factor 0 ≤ γ ≤ 1, policy, partial observability (POMDP), pole-balancing example, name from Andrey Markov.
- Wikipedia — Reinforcement learning — agent–environment interaction in discrete time steps, return G = R₁ + γR₂ + γ²R₃ + …
- [Sutton & Barto — Reinforcement Learning: An Introduction, 2nd edition](http://incompleteideas.net/book/the-book-2nd.html) — finite MDPs, returns and episodes.
- Arduino Servo library (GitHub) — controlling RC hobby servos.
- Photos: Wikimedia Commons (credits under each image).
Self-attested for now — keep artifacts in your lab notebook.
Complete
Today's goal
0 of 1 done today
Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).