Stage 14 · Lesson 02/10

Advanced Robotics / Reinforcement Learning · Lesson 02

States, Actions & Rewards

Est. 60–75 min

Why

Why this lesson matters

Before a robot can learn anything with reinforcement learning, a human must design the problem: what the robot is allowed to see (the state), what it is allowed to do (the actions), what counts as good (the reward), and when an attempt is over (the episode). This design is called a Markov decision process (MDP). A badly designed MDP makes learning slow or impossible, no matter how clever the algorithm.

In this lesson you will build a small desk-pick task as a Python environment with the same reset() / step() shape used by Gymnasium, the standard RL toolkit. You will see why leaving out one sensor changes the result, what γ (the discount) does to a return, and how terminated differs from truncated. On the bench, the UNO will log its own (s, a, r, s′) lines as CSV.

Concept

The Markov decision process, piece by piece

1. The four parts of an MDP

An MDP is usually written as the 4-tuple (S, A, P, R). The reward and the end-of-episode rules are design choices — you decide them.
SymbolNameMeaningIn our desk-pick task
SState spaceevery situation the robot can be in(gripper cell 0–4, jaw open/closed, holding yes/no) — 20 states
AAction spaceevery choice the robot can makeLEFT, RIGHT, CLOSE, OPEN, LIFT — 5 actions
P(s′ | s, a)Transitionwhat happens next, possibly randomCLOSE over the block grabs it 80 % of the time
RRewarda number after each step−1 per step, −3 lifting empty, +10 lifting the block
Diagram with three green circles labelled S0, S1, S2 and smaller orange circles for actions a0 and a1, connected by arrows marked with probabilities like 0.7 and 0.3, and two wavy arrows marked +5 and −1
A small MDP drawn as a graph: green circles are states, orange circles are actions, numbers on arrows are transition probabilities, and the wavy arrows are rewards (+5 and −1). Our desk-pick task is the same kind of picture, just with 20 states.Photo: MistWiz · Public domain · Source: Wikimedia Commons · resized, converted to WebP

2. The Markov property: the present is enough

The “Markov” in MDP means: the next state depends only on the current state and action, not on the full history. If you know where the gripper is, whether the jaw is closed and whether it is holding the block, you know everything needed to predict what LIFT will do. The name comes from the Russian mathematician Andrey Markov, who studied chains of random events with this property.

An old hand-drawn game board on yellowed paper with a grid of squares filled with handwritten script and long snakes drawn across it
A fragment of a hand-made Snakes and Ladders board from India (Wellcome Collection). The game is a perfect Markov chain: where you go next depends only on your current square and the dice — not on how you got there. Add a choice for the player and it becomes a decision process.Photo: Wellcome Collection · CC BY 4.0 · Source: Wikimedia Commons · resized, converted to WebP

3. Reward, return and discount

A reward is one number for one step. The return G is the total for the whole episode. Usually future rewards are multiplied by a discount factor γ (gamma) between 0 and 1: G = r₁ + γ·r₂ + γ²·r₃ + … With γ = 1 every step counts equally. With γ = 0.5 the robot is short-sighted: a reward 5 steps away is worth only 1/32 of its face value. In our run, the same successful episode has return +6.00 at γ = 1, +3.12 at γ = 0.9 and −1.25 at γ = 0.5 — the +10 at the end is discounted so much that the step costs win.

A small black wooden scorebox at a cricket ground with white number boards showing wickets 7, total 288 and overs 39
A cricket scoreboard: wickets 7, total 288, overs 39. The runs off one ball are like a reward; the total is like the return; and the board itself — score, wickets, overs — is a good state: it tells a captain everything needed to choose the next move.Photo: Acabashi · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

4. Episodes: terminated vs truncated

An episode is one attempt from reset() until it ends. Gymnasium’s step() returns two separate flags. terminated means the task itself ended — success (block lifted) or a real failure (fell into lava, robot crashed). truncated means we stopped for an outside reason, usually a time limit. Learning algorithms treat them differently: after “terminated” there is no future reward; after “truncated” the future simply was not observed. Older Gym code had one done flag; Gymnasium replaced it with these two because mixing them up confuses learning.

The Gymnasium step shape (what every RL environment returns) · Python
obs, info = env.reset(seed=42)
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
    obs, info = env.reset()

Visual

Visualise it: trace one transition

Pick a state and action in a toy transition table and read the reward and next state. This is exactly what env.step() does. (Educational simulation.)

Interactive · RL MDP explorer · Educational

Browser toy only — NOT MuJoCo / Isaac Gym / stable-baselines / industrial RL. Discrete teaching MDP for a pick skill.

State s

Approach

Last reward r

—

Episode return

0.00 · 0 steps

Actions a (from current state)

Pick a state, then take an action. Watch reward + next state.

ApproachAt graspLiftDone
Toy transition table (partial)
sars′
ApproachMove closer-0.10At grasp
ApproachClose gripper-1.00Fail
ApproachRaise-0.20Approach
ApproachRetreat-0.05Approach
At graspMove closer-0.05At grasp
At graspClose gripper0.50Lift
At graspRaise-1.00Fail
At graspRetreat-0.10Approach
LiftMove closer-0.05Lift
LiftClose gripper-0.02Lift
LiftRaise1.00Done
LiftRetreat-0.50Fail
DoneMove closer0.00Done
DoneClose gripper0.00Done
DoneRaise0.00Done
DoneRetreat0.00Done

Code lab

Code lab: build a desk-pick environment

desk_pick_env.py writes the desk-pick task as a class with reset(seed) and step(action) returning (obs, reward, terminated, truncated, info) — the same shape as gymnasium.Env, but with only the Python standard library so you can run it on any laptop. It then tests three hand-written policies (no learning yet) on the same 500 seeds.

desk_pick_env.py · Python
"""desk_pick_env.py - Stage 14 L02: a robot task written as an MDP.

A Gymnasium-style environment (same reset/step shape as gymnasium.Env),
written with the standard library only so it runs anywhere.

The desk is a line of 5 cells. The block sits in cell 3.
State  : (gripper cell 0..4, jaw "open"/"closed", holding True/False)
Actions: 0 LEFT, 1 RIGHT, 2 CLOSE, 3 OPEN, 4 LIFT
Reward : -1 every step (time costs), -3 for lifting an empty gripper,
         +10 for lifting the block (success, episode terminated).
Physics: closing over the block grabs it only 80 % of the time (slip).
Episode is truncated after 20 steps (a time limit, not a task outcome).
"""
import random

LEFT, RIGHT, CLOSE, OPEN, LIFT = range(5)
NAMES = ["LEFT", "RIGHT", "CLOSE", "OPEN", "LIFT"]


class DeskPickEnv:
    def __init__(self, block_cell=3, grip_success=0.8, max_steps=20, show_holding=True):
        self.block_cell = block_cell
        self.grip_success = grip_success
        self.max_steps = max_steps
        self.show_holding = show_holding   # False = the sensor that tells "holding" is missing
        self.rng = random.Random()

    def _obs(self):
        holding = self.holding if self.show_holding else None
        return (self.cell, "closed" if self.closed else "open", holding)

    def reset(self, seed=None):
        if seed is not None:
            self.rng.seed(seed)
        self.cell, self.closed, self.holding, self.t = 0, False, False, 0
        return self._obs(), {"block_cell": self.block_cell}

    def step(self, action):
        assert action in range(5), "action outside the action space"
        self.t += 1
        reward, terminated = -1.0, False
        if action == LEFT and not self.closed:
            self.cell = max(0, self.cell - 1)
        elif action == RIGHT and not self.closed:
            self.cell = min(4, self.cell + 1)
        elif action == CLOSE and not self.closed:
            self.closed = True
            self.holding = (self.cell == self.block_cell and self.rng.random() < self.grip_success)
        elif action == OPEN:
            self.closed, self.holding = False, False
        elif action == LIFT:
            if self.holding:
                reward, terminated = 10.0, True
            else:
                reward = -3.0
                self.closed = False           # arm comes back up and re-opens
        truncated = (not terminated) and self.t >= self.max_steps
        return self._obs(), reward, terminated, truncated, {"t": self.t}


# ---- three policies: rules that map an observation to an action ----
def random_policy(obs, rng):
    return rng.randrange(5)


def careful_policy(obs, rng):
    """Uses the 'holding' sensor: re-grip if the first grip slipped."""
    cell, jaw, holding = obs
    if cell < 3:
        return RIGHT if jaw == "open" else OPEN
    if jaw == "open":
        return CLOSE
    return LIFT if holding else OPEN


def blind_policy(obs, rng):
    """No holding sensor: always close then lift and hope."""
    cell, jaw, _ = obs
    if cell < 3:
        return RIGHT if jaw == "open" else OPEN
    return CLOSE if jaw == "open" else LIFT


def run_episode(env, policy, seed, gamma=1.0, trace=False):
    obs, info = env.reset(seed=seed)
    prng = random.Random(seed + 1000)
    g, discount, steps = 0.0, 1.0, 0
    while True:
        a = policy(obs, prng)
        nobs, r, term, trunc, info = env.step(a)
        if trace:
            print(f"   t={info['t']:2d}  s={obs}  a={NAMES[a]:5s}  r={r:+5.1f}  s'={nobs}"
                  + ("  TERMINATED" if term else "") + ("  TRUNCATED" if trunc else ""))
        g += discount * r
        discount *= gamma
        steps += 1
        obs = nobs
        if term or trunc:
            return g, term, steps


def main():
    env = DeskPickEnv()
    print("1) One episode with the careful policy (seed 4) - every row is (s, a, r, s')")
    g, term, steps = run_episode(env, careful_policy, seed=4, trace=True)
    print(f"   return = {g:+.1f} in {steps} steps")
    print("\n1b) Same policy, seed 0 - the first grip slips, the holding sensor says False")
    g, term, steps = run_episode(env, careful_policy, seed=0, trace=True)
    print(f"   return = {g:+.1f} in {steps} steps")

    print("\n2) 500 episodes per policy (same seeds for a fair test)")
    for name, pol, show in [("random", random_policy, True),
                            ("careful (sees holding)", careful_policy, True),
                            ("blind (no holding sensor)", blind_policy, False)]:
        env = DeskPickEnv(show_holding=show)
        res = [run_episode(env, pol, seed=s) for s in range(500)]
        avg = sum(r[0] for r in res) / len(res)
        succ = sum(1 for r in res if r[1])
        print(f"   {name:27s} avg return {avg:+6.2f}  success {succ:3d}/500  avg steps {sum(r[2] for r in res)/500:5.1f}")

    print("\n3) Same careful episode, different discount gamma")
    for gamma in (1.0, 0.9, 0.5):
        g, _, _ = run_episode(DeskPickEnv(), careful_policy, seed=4, gamma=gamma)
        print(f"   gamma {gamma:.1f}: discounted return {g:+.2f}")

    print("\n4) Terminated vs truncated (random policy, 500 episodes)")
    env = DeskPickEnv()
    res = [run_episode(env, random_policy, seed=s) for s in range(500)]
    print(f"   terminated (task really ended)  : {sum(1 for r in res if r[1])}")
    print(f"   truncated  (20-step time limit) : {sum(1 for r in res if not r[1])}")

    print("\n5) Size of this MDP")
    states = 5 * 2 * 2
    print(f"   states {states} (5 cells x jaw x holding), actions 5, table entries {states * 5}")


if __name__ == "__main__":
    main()

Output (run with python3 desk_pick_env.py)

1) One episode with the careful policy (seed 4) - every row is (s, a, r, s')
   t= 1  s=(0, 'open', False)  a=RIGHT  r= -1.0  s'=(1, 'open', False)
   t= 2  s=(1, 'open', False)  a=RIGHT  r= -1.0  s'=(2, 'open', False)
   t= 3  s=(2, 'open', False)  a=RIGHT  r= -1.0  s'=(3, 'open', False)
   t= 4  s=(3, 'open', False)  a=CLOSE  r= -1.0  s'=(3, 'closed', True)
   t= 5  s=(3, 'closed', True)  a=LIFT   r=+10.0  s'=(3, 'closed', True)  TERMINATED
   return = +6.0 in 5 steps

1b) Same policy, seed 0 - the first grip slips, the holding sensor says False
   t= 1  s=(0, 'open', False)  a=RIGHT  r= -1.0  s'=(1, 'open', False)
   t= 2  s=(1, 'open', False)  a=RIGHT  r= -1.0  s'=(2, 'open', False)
   t= 3  s=(2, 'open', False)  a=RIGHT  r= -1.0  s'=(3, 'open', False)
   t= 4  s=(3, 'open', False)  a=CLOSE  r= -1.0  s'=(3, 'closed', False)
   t= 5  s=(3, 'closed', False)  a=OPEN   r= -1.0  s'=(3, 'open', False)
   t= 6  s=(3, 'open', False)  a=CLOSE  r= -1.0  s'=(3, 'closed', True)
   t= 7  s=(3, 'closed', True)  a=LIFT   r=+10.0  s'=(3, 'closed', True)  TERMINATED
   return = +4.0 in 7 steps

2) 500 episodes per policy (same seeds for a fair test)
   random                      avg return -26.95  success  22/500  avg steps  19.8
   careful (sees holding)      avg return  +5.51  success 500/500  avg steps   5.5
   blind (no holding sensor)   avg return  +5.02  success 500/500  avg steps   5.5

3) Same careful episode, different discount gamma
   gamma 1.0: discounted return +6.00
   gamma 0.9: discounted return +3.12
   gamma 0.5: discounted return -1.25

4) Terminated vs truncated (random policy, 500 episodes)
   terminated (task really ended)  : 22
   truncated  (20-step time limit) : 478

5) Size of this MDP
   states 20 (5 cells x jaw x holding), actions 5, table entries 100

Try it: pick a state and an action, then read the reward and next state from the toy transition table. Write down three (s, a, r, s′) rows by hand. (Educational simulation — not MuJoCo, Isaac or a real RL library.)

Interactive · RL MDP explorer · Educational

Browser toy only — NOT MuJoCo / Isaac Gym / stable-baselines / industrial RL. Discrete teaching MDP for a pick skill.

State s

Approach

Last reward r

—

Episode return

0.00 · 0 steps

Actions a (from current state)

Pick a state, then take an action. Watch reward + next state.

ApproachAt graspLiftDone
Toy transition table (partial)
sars′
ApproachMove closer-0.10At grasp
ApproachClose gripper-1.00Fail
ApproachRaise-0.20Approach
ApproachRetreat-0.05Approach
At graspMove closer-0.05At grasp
At graspClose gripper0.50Lift
At graspRaise-1.00Fail
At graspRetreat-0.10Approach
LiftMove closer-0.05Lift
LiftClose gripper-0.02Lift
LiftRaise1.00Done
LiftRetreat-0.50Fail
DoneMove closer0.00Done
DoneClose gripper0.00Done
DoneRaise0.00Done
DoneRetreat0.00Done

Bench build: the UNO writes its own transition log

transition_logger.ino reuses the Lesson 1 light-seeking head but now speaks MDP. State = (angle bin 0–8, light level 0–3). Actions = LEFT, STAY, RIGHT by 15°. Reward = +1 if the light level went up, −1 if down, −0.1 otherwise. The episode terminates at light level 3 and is truncated after 30 steps. The actions are random on purpose — today we only collect data. Lesson 3 will learn from it.

Safety: never power the servo from a GPIO pin or the UNO 5 V pin — use its own supply with grounds joined. Keep the torch steady on the bench (a clamp or stand), not held near someone’s face.
PartConnectionNote
SG90/MG90-class servosignal → D9LDR taped to the horn, shaded by a paper tube
Servo supplyred → separate 5–6 V, brown → supply GNDjoin supply GND to UNO GND
LDR + 10 kΩLDR 5 V → A0, 10 kΩ A0 → GNDmeasure dark/bright readings, then edit LEVEL_EDGES
Push buttonD2 → button → GNDINPUT_PULLUP; press to start each episode (env.reset)
transition_logger.ino · Arduino C++
// transition_logger.ino - Stage 14 L02
// The L01 light-seeker head, now written as an MDP and logged as (s, a, r, s').
// State  s = (angle bin 0..8, light level 0..3)
// Action a = 0 LEFT (-15 deg), 1 STAY, 2 RIGHT (+15 deg)   - chosen at random here
// Reward r = +1 if the light level went up, -1 if it went down, -0.1 otherwise (time cost)
// Episode: TERMINATED when light level 3 is reached, TRUNCATED after 30 steps.
//
// Wiring: same as L01. Servo signal D9 (servo on its OWN 5-6 V supply, grounds joined),
// LDR from 5V to A0 + 10k from A0 to GND, button D2 -> GND starts a new episode.
// Serial 115200 prints CSV you can paste into a spreadsheet or Python.
#include <Servo.h>

const int SERVO_PIN = 9, LDR_PIN = A0, BUTTON_PIN = 2;
const int MAX_STEPS = 30;
// Light thresholds: measure YOUR room first (see lesson) and edit these.
const int LEVEL_EDGES[3] = {300, 550, 800};

Servo head;
int angleBin = 4;            // bin 4 = 90 degrees
int stepCount = 0;
int episode = 0;
bool running = false;

int binToAngle(int b) { return 30 + 15 * b; }   // bins 0..8 -> 30..150 deg

int readLevel() {
  long sum = 0;
  for (int i = 0; i < 8; i++) { sum += analogRead(LDR_PIN); delay(5); }
  int raw = sum / 8;
  int level = 0;
  while (level < 3 && raw >= LEVEL_EDGES[level]) level++;
  return level;
}

void moveToBin(int b) {
  int from = head.read(), to = binToAngle(b);
  int dir = (to > from) ? 1 : -1;
  for (int a = from; a != to; a += dir) { head.write(a); delay(10); }
  head.write(to);
  delay(350);                // settle before reading the sensor
}

void startEpisode() {
  episode++;
  stepCount = 0;
  angleBin = random(9);      // random start state, like env.reset()
  moveToBin(angleBin);
  running = true;
  Serial.print(F("# episode ")); Serial.println(episode);
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  randomSeed(analogRead(A5));
  head.attach(SERVO_PIN);
  head.write(90);
  delay(500);
  Serial.println(F("episode,t,s_bin,s_light,a,r,s2_bin,s2_light,terminated,truncated"));
  Serial.println(F("# press the button to start an episode"));
}

void loop() {
  if (!running) {
    if (digitalRead(BUTTON_PIN) == LOW) { delay(30); startEpisode(); }
    return;
  }
  int sBin = angleBin, sLight = readLevel();
  int a = random(3);                         // 0 LEFT, 1 STAY, 2 RIGHT
  int nextBin = constrain(sBin + (a - 1), 0, 8);   // walls: cannot go past 30 or 150 deg
  if (nextBin != sBin) moveToBin(nextBin);
  angleBin = nextBin;
  int s2Light = readLevel();
  stepCount++;

  float r = -0.1;
  if (s2Light > sLight) r = 1.0;
  else if (s2Light < sLight) r = -1.0;
  bool terminated = (s2Light == 3);
  bool truncated = !terminated && stepCount >= MAX_STEPS;

  Serial.print(episode); Serial.print(',');
  Serial.print(stepCount); Serial.print(',');
  Serial.print(sBin); Serial.print(','); Serial.print(sLight); Serial.print(',');
  Serial.print(a); Serial.print(','); Serial.print(r, 1); Serial.print(',');
  Serial.print(angleBin); Serial.print(','); Serial.print(s2Light); Serial.print(',');
  Serial.print(terminated ? 1 : 0); Serial.print(','); Serial.println(truncated ? 1 : 0);

  if (terminated || truncated) {
    running = false;
    Serial.println(terminated ? F("# TERMINATED: reached level 3") : F("# TRUNCATED: 30-step limit"));
    Serial.println(F("# press the button for the next episode"));
  }
  delay(150);
}
Serial Monitor (115200) — format of the log (numbers depend on your room) · Text
episode,t,s_bin,s_light,a,r,s2_bin,s2_light,terminated,truncated
# press the button to start an episode
# episode 1
1,1,<bin>,<level>,<0-2>,<-0.1 / 1.0 / -1.0>,<bin>,<level>,0,0
...
# TERMINATED: reached level 3      (or)      # TRUNCATED: 30-step limit

Debugging

SymptomLikely causeFix
Every episode is TRUNCATEDlevel 3 threshold too high for your torchMeasure the bright reading and lower LEVEL_EDGES[2]
Every episode TERMINATES at step 1room already brighter than level 3Raise the thresholds or dim the room
Same (s, a) gives different s′ oftenservo still shaking during reading, or flickering tube lightIncrease the 350 ms settle delay; use a steady LED torch
Python: step() after the episode ended gives strange resultsforgot to call reset()Always reset when terminated or truncated is True
Python: return looks wronggamma multiplied in the wrong orderMultiply the reward by the current discount, then update discount *= gamma
UNO resets when the servo movesservo on UNO 5 VSeparate supply, grounds joined

Real world

MDPs in the real world

A small two-wheeled robot chassis with yellow wheels and a wooden base on a table
A small two-wheeled robot base. If you wanted it to learn line following, the state could be the line-sensor readings and wheel speeds, the actions a few steering commands, and the reward “stay on the line and keep moving”.Photo: Gsparr86 · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP
  • Pole balancing (CartPole): a classic MDP example. The state describes where the cart and pole are and how fast they move; the actions push the cart left or right; the episode ends when the pole falls.
  • Robot arms: state is often joint angles and speeds plus the object’s pose from the camera (Stage 13); actions are small joint or gripper commands.
  • Mobile robots: state = pose and nearby obstacles (from an occupancy map); actions = velocity commands; reward = progress minus collisions.
  • Design is the hard part: the Gymnasium guide itself warns that a poorly designed environment makes learning difficult or impossible no matter how good the algorithm is.
Black-and-white portrait photograph of an older man with a beard and glasses wearing a dark suit
Andrey Markov (1856–1922), the mathematician whose chains of random events gave the Markov property — and the MDP — its name.Photo: Unknown author · Public domain · Source: Wikimedia Commons · resized, converted to WebP

Think

Think about it

Think

You design a state for the light-seeker as just “light level 0–3”, without the angle. Is that a good state? What goes wrong?

Experiment

Try it in the lab

About 60 minutes. Python 3 for the desk-pick environment; the Lesson 1 light-seeker hardware for the logger.

Experiment

Design, run and log an MDP

You will need

  • `desk_pick_env.py` (Python 3, no libraries)
  • Toy MDP explorer (in your browser)
  • Arduino UNO, servo on a separate 5–6 V supply, LDR + 10 kΩ, push button, jumper wires
  • Steady LED torch on a stand, spreadsheet software

Steps

  1. Run `desk_pick_env.py`. Copy the seed-0 trace into your notebook as a table with columns s, a, r, s′.
  2. Change `grip_success` from 0.8 to 0.5. Run again. How much bigger does the gap between the careful and blind policies get?
  3. Change the empty-lift penalty from −3 to −20. Does any policy’s average return change much? Why?
  4. On the bench, calibrate LEVEL_EDGES for your room, then run 5 episodes of the logger and save the CSV.
  5. In the spreadsheet, count how many episodes terminated vs were truncated. Find one (s, a) pair that gave two different s′.
  6. In the MDP explorer, find the action that ends an episode with the worst reward and the shortest path to the best reward.

Interactive · RL MDP explorer · Educational

Browser toy only — NOT MuJoCo / Isaac Gym / stable-baselines / industrial RL. Discrete teaching MDP for a pick skill.

State s

Approach

Last reward r

—

Episode return

0.00 · 0 steps

Actions a (from current state)

Pick a state, then take an action. Watch reward + next state.

ApproachAt graspLiftDone
Toy transition table (partial)
sars′
ApproachMove closer-0.10At grasp
ApproachClose gripper-1.00Fail
ApproachRaise-0.20Approach
ApproachRetreat-0.05Approach
At graspMove closer-0.05At grasp
At graspClose gripper0.50Lift
At graspRaise-1.00Fail
At graspRetreat-0.10Approach
LiftMove closer-0.05Lift
LiftClose gripper-0.02Lift
LiftRaise1.00Done
LiftRetreat-0.50Fail
DoneMove closer0.00Done
DoneClose gripper0.00Done
DoneRaise0.00Done
DoneRetreat0.00Done

An MDP is a design: choosing the state, actions, reward and episode rules decides what the robot can possibly learn. Missing information (partial observability) shows up as wasted actions and inconsistent transitions in the log.

Knowledge check

Check your understanding

Knowledge check

1. An MDP is usually written as the tuple…

2. The Markov property says the next state depends on…

3. With rewards −1, −1, +10 and γ = 0.9, the return is…

4. In Gymnasium, `truncated = True` usually means…

5. The blind policy without the holding sensor had a lower return because…

6. What must you call after `terminated` or `truncated` is True?

7. A very small γ (like 0.5) makes the agent…

8. In the logger, which part is NOT a design choice made by you?

Mini assignment

Hands-on assignment

Mini assignment

MDP card for your own robot task

Write a real robot task as a clean MDP and test it.

Deliverables

  • An MDP card for a desk task of your choice: S, A, P (what is random?), R, γ, and the terminated / truncated rules.
  • Your hand-made (s, a, r, s′) table for the seed-0 desk-pick episode, with the return computed at γ = 1 and γ = 0.9.
  • The Python result with grip_success 0.5: careful vs blind average return, and your explanation.
  • The UNO CSV for 5 episodes, with terminated/truncated counts and one example of an inconsistent transition.

Do this, step by step

  1. Write your MDP card: S, A, P, R, γ, and when an episode is terminated or truncated.
  2. Run desk_pick_env.py and fill the (s, a, r, s′) table for the seed-0 episode by hand.
  3. Change grip_success from 0.8 to 0.5 and compare the careful and blind policies.
  4. Bench: upload transition_logger.ino and save 5 episodes of CSV from the Serial Monitor.
  5. Count terminated vs truncated (30 steps, MAX_STEPS) and find one transition that looks inconsistent.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. Return at γ = 0.9: multiply the reward of step k by 0.9^k before adding — the first step uses 0.9^0 = 1.
  2. If every episode is truncated, your room never reaches light level 3 — lower LEVEL_EDGES {300, 550, 800} to match your LDR readings.
  3. An “inconsistent” transition is the same (s, a) giving a different s′ — that is the randomness P describes, not a bug.

Challenges (pick by level)

  • 🟢 Beginner: Write Snakes and Ladders as an MDP. What is the state? What are the actions (hint: there are none — so what is it?). Now add a rule “you may choose to roll one dice or two”. What changes?
  • 🟡 Intermediate: Add a second block in cell 1 to desk_pick_env.py. Change the state so the policy can know which block it holds. How many states now?
  • 🟠 Advanced: Add render() to the environment that prints the desk as text, like . . . [B] . with the gripper marked, like the simple text render in the Gymnasium custom-environment tutorial.
  • 🔴 Expert: Install Gymnasium (pip install gymnasium), turn DeskPickEnv into a real gymnasium.Env subclass with Discrete action and observation spaces, and run gymnasium.utils.env_checker.check_env on it.
  • ⚫ Research: Read the Wikipedia article on Markov decision processes and find the name of the version where the state is not fully visible. Describe one robot task in your lab that is really of that kind.

Next step

Our policies today were written by hand. The next question is: how good is each state, and how should the robot choose actions to get the highest return? In Lesson 3 — Policies & Value you will compute value tables and read a greedy policy from them.

Look back: Lesson 1: Why Robots Learn. Next: Lesson 3: Policies & Value. Related: Stage 13 — Perception, Planning & Manipulation.

Sources

  • Gymnasium — Env API (Farama Foundation) — reset(seed) returns (observation, info); step(action) returns (observation, reward, terminated, truncated, info); terminated vs truncated and the removal of done in v0.26.
  • Gymnasium — Create a Custom Environment — environment design questions, GridWorld example, Discrete(4) actions, reward and state-representation pitfalls, check_env.
  • Wikipedia — Markov decision process — the 4-tuple (S, A, Pₐ, Rₐ), discount factor 0 ≤ γ ≤ 1, policy, partial observability (POMDP), pole-balancing example, name from Andrey Markov.
  • Wikipedia — Reinforcement learning — agent–environment interaction in discrete time steps, return G = R₁ + γR₂ + γ²R₃ + …
  • [Sutton & Barto — Reinforcement Learning: An Introduction, 2nd edition](http://incompleteideas.net/book/the-book-2nd.html) — finite MDPs, returns and episodes.
  • Arduino Servo library (GitHub) — controlling RC hobby servos.
  • Photos: Wikimedia Commons (credits under each image).

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in