Stage 11 · Lesson 01/16

Computer Vision for Robotics (Vision Foundations) · Lesson 01

Why Robots Need Vision

Est. 45–60 min

Why

Why this lesson matters

Think of a cook in a dark kitchen during a power cut. She knows exactly where her own hands are — she can touch her nose with her eyes closed. But she cannot find the salt dabba if someone moved it. The moment the inverter light comes on, she sees the dabba and picks it up in one go. A robot arm is the same. Encoders and potentiometers tell it about its own body. A camera tells it about the world — where the block is, whether it is there at all, whether a hand is in the way.

Stages 7–10 made the arm accurate, smooth and well organised. Stage 11 makes it aware. In this first lesson you will compare a blind arm and a seeing arm in a small Python simulation, look through a real webcam with OpenCV and see that the robot gets only numbers, and build a two-sensor Arduino board that shows the difference between body sense and world sense.

Concept

Body sense vs world sense

1. Two kinds of sensing

Proprioception is the sense of self-movement and body position. Robots need both kinds of sensing to do useful work in a changing world.
Proprioception (body sense)Exteroception (world sense)
Question it answers“Where are my joints? How fast am I moving?”“What is around me? Where is the object?”
Sensors on our armservo/pot angles, encoders, IMUcamera, depth camera, ultrasonic, light sensor
In ROS 2 (Stage 10)/joint_statesimage topics, point clouds, /target_pose
If it is missingthe arm cannot control its own motionthe arm moves perfectly to the wrong place

2. What a camera really gives a robot

We see a red block. The robot gets a grid of numbers. A colour image of 640 × 480 pixels is 480 rows × 640 columns × 3 colour values, each from 0 to 255. Nothing in that grid says “block”. Computer vision is the work of turning that grid into a useful answer: “red block, centre at (23.8 cm, 9.8 cm), sure enough to pick”.

3. The vision pipeline you will build in Stage 11

  1. Capture

    The camera gives a frame: an array of pixels (Lessons 2–3).

  2. Clean up

    Grayscale, blur and threshold remove noise and lighting changes (Lesson 4).

  3. Find

    Edges, shapes, features, colours or a trained detector find the object in pixels (Lessons 5–11).

  4. Measure

    Calibration, depth and pose turn pixels into centimetres and angles in the robot’s frame (Lessons 12–15).

  5. Act

    IK, trajectories and the ROS 2 system from Stages 8–10 move the arm to the object (Lesson 16 capstone).

Hand sketch on lined paper of a robot arm on a wheeled base with joint angles theta 1, theta 2 and theta 3, reaching with a curved gripper towards a box
A student’s sketch of a mobile arm reaching for a box. The arm knows θ1, θ2 and θ3 from its own sensors — but only a sensor that looks at the world can tell it where the box is today.Photo: Ckalafatis24 · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP

Visual

Visualise it: a robot sees numbers first

Educational pixel explorer: a synthetic scene, its (u, v) coordinates and RGB values — robots see numbers first. (A browser sim, not OpenCV.)

Interactive · Image pixel explorer · Educational

Educational robot vision pipeline: camera to actionPixel grid diagram

Hover the synthetic camera image. Robots see numbers, not “red block” — until algorithms assign meaning.

u

—

v

—

R

—

G

—

B

—

Browser canvas educational sim — not OpenCV.js / not a live camera pipeline.

Code lab

Code lab: a blind arm vs a seeing arm

blind_vs_seeing_arm.py needs only Python and NumPy. It makes a tiny top-down “camera image” (64 × 48 pixels, 0.5 cm per pixel) of a light-wood table with a red block. The blind arm goes to the spot it was taught. The seeing arm finds the red pixels, takes their centre, converts pixels to centimetres and goes there. Five situations are tested, including a missing block and dim lights. Run python3 blind_vs_seeing_arm.py.

blind_vs_seeing_arm.py · Python
"""blind_vs_seeing_arm.py - Stage 11 L01: why a robot arm needs a camera.

Two arms try to pick a red block from a table:
  BLIND arm  - goes to the spot it remembers (perfect kinematics, no camera)
  SEEING arm - looks at a top-down camera image, finds red pixels, then goes there
The camera image is a tiny synthetic picture made with NumPy (no OpenCV needed).
Camera model is deliberately simple: looking straight down, 0.5 cm per pixel,
pixel (0, 0) = table corner (0, 0). Real cameras need calibration (later lessons).
"""
import numpy as np

W, H = 64, 48            # image size in pixels (u across, v down)
CM_PER_PX = 0.5          # 64 px x 0.5 = 32 cm wide table view
GRIP_TOL_CM = 1.5        # gripper can only absorb this much error
REMEMBERED = (20.0, 10.0)  # where the block was when we "taught" the arm, cm


def make_image(block_cm=None, light=1.0):
    """Light-wood table, optional 3 x 3 cm red block centred at block_cm."""
    img = np.zeros((H, W, 3), dtype=np.float32)
    img[:, :] = (200, 170, 130)                       # table colour (R, G, B)
    if block_cm is not None:
        u0, v0 = int(block_cm[0] / CM_PER_PX), int(block_cm[1] / CM_PER_PX)
        img[v0 - 3:v0 + 3, u0 - 3:u0 + 3] = (210, 30, 30)  # 6 x 6 px = 3 x 3 cm
    return np.clip(img * light, 0, 255).astype(np.uint8)


def find_red(img):
    """Return (u, v) centroid of 'red' pixels, or None. A fixed threshold, on purpose."""
    r, g, b = img[:, :, 0].astype(int), img[:, :, 1].astype(int), img[:, :, 2].astype(int)
    mask = (r > 150) & (g < 80) & (b < 80)
    if mask.sum() < 10:                     # fewer than 10 pixels = not a block
        return None, int(mask.sum())
    vs, us = np.nonzero(mask)
    return (us.mean(), vs.mean()), int(mask.sum())


def ascii_view(img, step=4):
    rows = []
    for v in range(0, H, step):
        row = ''
        for u in range(0, W, step // 2):
            r, g, b = img[v, u]
            row += '#' if r > 150 and g < 80 and b < 80 else '.'
        rows.append('    ' + row)
    return '\n'.join(rows)


def try_pick(name, block_cm, light=1.0, show=False):
    print(f'\n=== {name} ===')
    img = make_image(block_cm, light)
    if show:
        print('  what the camera sees (# = red pixel):')
        print(ascii_view(img))
        print(f'  raw numbers at pixel (u=40, v=20): RGB = {tuple(int(x) for x in img[20, 40])}')
    # blind arm
    if block_cm is None:
        print(f'  BLIND : closes gripper at {REMEMBERED} cm on empty air -> MISS (and it does not know)')
    else:
        err = np.hypot(block_cm[0] - REMEMBERED[0], block_cm[1] - REMEMBERED[1])
        verdict = 'PICK OK' if err <= GRIP_TOL_CM else 'MISS'
        print(f'  BLIND : goes to {REMEMBERED} cm, block is {block_cm} cm, error {err:.1f} cm -> {verdict}')
    # seeing arm
    c, n = find_red(img)
    if c is None:
        print(f'  SEEING: only {n} red pixels found -> "no block seen", refuses to move')
        return
    x, y = c[0] * CM_PER_PX, c[1] * CM_PER_PX
    err = np.hypot(x - block_cm[0], y - block_cm[1])
    print(f'  SEEING: {n} red pixels, centroid (u={c[0]:.1f}, v={c[1]:.1f}) px'
          f' -> ({x:.1f}, {y:.1f}) cm, error {err:.1f} cm -> '
          + ('PICK OK' if err <= GRIP_TOL_CM else 'MISS'))


if __name__ == '__main__':
    try_pick('1. Block exactly where it was taught', (20.0, 10.0), show=True)
    try_pick('2. Cleaner moved the block 4 cm', (24.0, 10.0))
    try_pick('3. Block placed somewhere new', (8.0, 18.0), show=True)
    try_pick('4. Block missing', None)
    try_pick('5. Same block, lights dimmed to 40%', (20.0, 10.0), light=0.4)

Output (run with python3 blind_vs_seeing_arm.py)


=== 1. Block exactly where it was taught ===
  what the camera sees (# = red pixel):
    ................................
    ................................
    ................................
    ................................
    ................................
    ...................###..........
    ................................
    ................................
    ................................
    ................................
    ................................
    ................................
  raw numbers at pixel (u=40, v=20): RGB = (210, 30, 30)
  BLIND : goes to (20.0, 10.0) cm, block is (20.0, 10.0) cm, error 0.0 cm -> PICK OK
  SEEING: 36 red pixels, centroid (u=39.5, v=19.5) px -> (19.8, 9.8) cm, error 0.4 cm -> PICK OK

=== 2. Cleaner moved the block 4 cm ===
  BLIND : goes to (20.0, 10.0) cm, block is (24.0, 10.0) cm, error 4.0 cm -> MISS
  SEEING: 36 red pixels, centroid (u=47.5, v=19.5) px -> (23.8, 9.8) cm, error 0.4 cm -> PICK OK

=== 3. Block placed somewhere new ===
  what the camera sees (# = red pixel):
    ................................
    ................................
    ................................
    ................................
    ................................
    ................................
    ................................
    ................................
    ................................
    .......###......................
    ................................
    ................................
  raw numbers at pixel (u=40, v=20): RGB = (200, 170, 130)
  BLIND : goes to (20.0, 10.0) cm, block is (8.0, 18.0) cm, error 14.4 cm -> MISS
  SEEING: 36 red pixels, centroid (u=15.5, v=35.5) px -> (7.8, 17.8) cm, error 0.4 cm -> PICK OK

=== 4. Block missing ===
  BLIND : closes gripper at (20.0, 10.0) cm on empty air -> MISS (and it does not know)
  SEEING: only 0 red pixels found -> "no block seen", refuses to move

=== 5. Same block, lights dimmed to 40% ===
  BLIND : goes to (20.0, 10.0) cm, block is (20.0, 10.0) cm, error 0.0 cm -> PICK OK
  SEEING: only 0 red pixels found -> "no block seen", refuses to move

Try it: hover over the scene and read the (u, v) position and the RGB numbers. Meaning is not in the numbers yet — your code has to find it.

Interactive · Image pixel explorer · Educational

Educational robot vision pipeline: camera to actionPixel grid diagram

Hover the synthetic camera image. Robots see numbers, not “red block” — until algorithms assign meaning.

u

—

v

—

R

—

G

—

B

—

Browser canvas educational sim — not OpenCV.js / not a live camera pipeline.

first_look.py — look through a real camera

This script follows the official OpenCV “Getting Started with Videos” tutorial: cv.VideoCapture(0) opens the first camera, cap.isOpened() checks it, and cap.read() returns (ok, frame) for every frame. Once a second it prints the frame’s shape, the B, G, R values at the centre and the average brightness. Press s to save a snapshot and q to quit. With --image photo.jpg it gives the same report for a saved photo, so you can practise without a camera.

first_look.py · Python
"""first_look.py - Stage 11 L01: your robot's first look through a real camera (OpenCV).

  python3 first_look.py              # live webcam (device 0); keys: s = save snapshot, q = quit
  python3 first_look.py --image x.jpg   # same report for a saved photo (no camera needed)
Install:  pip install opencv-python
Prints what the robot really gets: an array of numbers (height x width x 3, in B, G, R order).
"""
import sys

import cv2 as cv


def report(frame):
    h, w = frame.shape[:2]
    b, g, r = frame[h // 2, w // 2]
    gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
    print(f'shape={frame.shape} dtype={frame.dtype} '
          f'centre BGR=({b},{g},{r}) mean brightness={gray.mean():.1f}/255')


def draw_crosshair(frame):
    h, w = frame.shape[:2]
    cv.line(frame, (w // 2 - 15, h // 2), (w // 2 + 15, h // 2), (0, 255, 0), 1)
    cv.line(frame, (w // 2, h // 2 - 15), (w // 2, h // 2 + 15), (0, 255, 0), 1)


def from_image(path):
    frame = cv.imread(path)          # returns None if the file cannot be read
    if frame is None:
        sys.exit(f'Cannot read {path}')
    report(frame)


def from_camera(index=0):
    cap = cv.VideoCapture(index)
    if not cap.isOpened():
        sys.exit('Cannot open camera - is another app using it? Try index 1.')
    print('size:', cap.get(cv.CAP_PROP_FRAME_WIDTH), 'x', cap.get(cv.CAP_PROP_FRAME_HEIGHT))
    n = 0
    while True:
        ok, frame = cap.read()
        if not ok:
            print("Can't receive frame. Exiting ...")
            break
        n += 1
        if n % 30 == 0:              # print about once a second at 30 fps
            report(frame)
        shown = frame.copy()
        draw_crosshair(shown)
        cv.imshow('first look (s = save, q = quit)', shown)
        key = cv.waitKey(1) & 0xFF
        if key == ord('s'):
            cv.imwrite('snapshot.png', frame)
            print('saved snapshot.png')
        elif key == ord('q'):
            break
    cap.release()
    cv.destroyAllWindows()


if __name__ == '__main__':
    if len(sys.argv) == 3 and sys.argv[1] == '--image':
        from_image(sys.argv[2])
    else:
        from_camera()
Run it · Shell
pip install opencv-python numpy
python3 first_look.py                 # webcam: s = save, q = quit
python3 first_look.py --image snapshot.png
# example line you will see (values depend on your room and camera):
# shape=(480, 640, 3) dtype=uint8 centre BGR=(30,30,210) mean brightness=173.9/255

Bench build: body sense and world sense on one Arduino

A potentiometer stands in for a joint encoder — it only knows its own angle. A light-dependent resistor (photocell) is a one-pixel camera — it knows nothing about the joint but notices when your hand casts a shadow. The photocell uses the voltage-divider method from the Adafruit photocell guide: one end to 5 V, the other end to the analog pin and a 10 kΩ resistor to ground, so more light gives a higher reading.

Pots and photocells draw well under a milliamp at 5 V, so they are safe on the Arduino 5 V pin.
PartConnectionNote
10 kΩ potentiometerends → 5 V and GND, wiper → A0the “joint encoder”
Photocell (LDR)5 V → LDR → A1the “one-pixel camera”
10 kΩ resistorA1 → resistor → GNDpull-down; use 1 kΩ in very bright rooms
On-board LEDD13 (built in)lights when a shadow falls on the LDR
Servos / motorsnot used hereif added later: separate 5–6 V supply, grounds joined, never from a GPIO
body_and_world.ino · Arduino C++
// body_and_world.ino - Stage 11 L01: "body sense" vs "world sense" on one Arduino UNO.
//   A0: 10 k potentiometer  = a joint encoder stand-in (proprioception: where is MY joint?)
//   A1: photocell (LDR) + 10 k resistor to GND = a one-pixel "camera" (exteroception: what is OUT THERE?)
//   D13: on-board LED lights when a hand/object shadows the LDR.
// Wiring (all low-current, safe on the Arduino 5 V pin):
//   pot ends -> 5 V and GND, wiper -> A0
//   5 V -> LDR -> A1 -> 10 k resistor -> GND   (Adafruit "pull-down" divider: brighter = higher reading)
// No motors here. If you later add a servo, power it from a separate supply, grounds joined.
const int POT_PIN = A0;
const int LDR_PIN = A1;
int baseline = 0;

void setup() {
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);
  long sum = 0;
  for (int i = 0; i < 32; i++) { sum += analogRead(LDR_PIN); delay(10); }
  baseline = sum / 32;                       // room light at power-on, keep the sensor uncovered
  Serial.print(F("baseline light = ")); Serial.println(baseline);
}

void loop() {
  int pot = analogRead(POT_PIN);             // 0..1023
  int light = analogRead(LDR_PIN);           // 0..1023
  float jointDeg = pot * (180.0 / 1023.0);   // pretend 0-180 deg joint, like a servo
  bool shadow = light < baseline * 6 / 10;   // 40% darker than the start = something in front
  digitalWrite(LED_BUILTIN, shadow ? HIGH : LOW);
  Serial.print(F("joint_deg=")); Serial.print(jointDeg, 1);
  Serial.print(F(" light=")); Serial.print(light);
  Serial.println(shadow ? F(" OBJECT?") : F(""));
  delay(100);
}

Debugging

SymptomLikely causeFix
“Cannot open camera”another app (video call, browser tab) is using it, or wrong indexClose other apps; try cv.VideoCapture(1); test with your system camera app
Window opens but stays blacklens cover, or camera still startingRemove the cover; skip the first few frames
Red object shows a high first numberyou are reading B, G, R as R, G, BRemember OpenCV order is B, G, R
cv.imshow error on a server or WSLno display availableUse --image mode or save frames with cv.imwrite
LDR reading stuck near 0 or 1023resistor missing or wired to the wrong railCheck: 5 V → LDR → A1 → 10 kΩ → GND
OBJECT? flickers all the timetube-light flicker or baseline taken while coveredKeep the LDR uncovered during reset; average more readings

Real world

Vision in real robots

  • Farm produce sorting: optical sorters such as the DOWNS CropVision use industrial cameras and an AI algorithm to look at the whole surface (360°) of every potato on the belt and send it to one of three exits — keep, second quality or waste — picking out clods, stones and cut, cracked or spoiled potatoes without a human checking each one.
  • Depth cameras for robots: small RGB-D cameras like the Intel RealSense D435 add a depth value to each pixel, so a robot knows not only where an object is in the picture but how far away it is (Lesson 14).
  • Your desk arm: an ordinary USB webcam fixed above the table is enough for the Stage 11 capstone: find the block, convert to centimetres, and send a target to the Stage 10 planner.
A large red and black DOWNS CropVision potato sorting machine on display at a farm machinery exhibition
A DOWNS CropVision optical potato sorter at Agritechnica 2023. Cameras plus software decide, potato by potato, what is good and what is waste.Photo: MB-one · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP
A small grey Intel RealSense D435 depth camera with several lenses, mounted on a mini tripod
An Intel RealSense D435 depth camera. Its extra lenses let it measure distance as well as colour — a common “eye” on research robots.Photo: Marc Auledas · CC BY-SA 4.0 · Source: Wikimedia Commons · resized, converted to WebP
A black Logitech C922 USB webcam on a small metal stand in front of a computer monitor
An ordinary USB webcam. This is all you need for most of Stage 11 — plus good, steady lighting.Photo: Zsinytwiki · CC0 · Source: Wikimedia Commons · resized, converted to WebP

Think

Think about it

Think

Your arm’s IK is perfect to 1 mm and the block is always placed in a 3D-printed holder on the table. Do you still need a camera? What if the holder is sometimes empty?

Experiment

Try it in the lab

About 45 minutes. Python part needs only NumPy; the camera part needs a webcam and OpenCV; the bench part needs an Arduino UNO, a 10 kΩ pot, a photocell and a 10 kΩ resistor.

Experiment

Blind, seeing and fooled

You will need

  • `blind_vs_seeing_arm.py`, Python 3 with NumPy
  • `first_look.py` with `opencv-python` and a webcam (or a saved photo)
  • Pixel explorer (in your browser)
  • Optional bench: Arduino UNO, 10 kΩ pot, photocell, 10 kΩ resistor, jumper wires

Steps

  1. Run `blind_vs_seeing_arm.py`. Then change `REMEMBERED` to (24, 10) and run again. Which cases change for the blind arm?
  2. Change `GRIP_TOL_CM` to 5. Does the blind arm now “pass” Case 2? Is that a good fix in real life?
  3. Change the light in Case 5 to 0.7, 0.6, 0.5. At what level does the seeing arm stop seeing? Why?
  4. In the pixel explorer, find the (u, v) of the red block’s centre and its RGB values.
  5. Run `first_look.py`. Write down the centre B, G, R for a red, a green and a white object.
  6. Bench: run `body_and_world.ino`, turn the pot and wave your hand. Then switch off the room light and watch the false `OBJECT?`.

Interactive · Image pixel explorer · Educational

Educational robot vision pipeline: camera to actionPixel grid diagram

Hover the synthetic camera image. Robots see numbers, not “red block” — until algorithms assign meaning.

u

—

v

—

R

—

G

—

B

—

Browser canvas educational sim — not OpenCV.js / not a live camera pipeline.

Body sense keeps the arm accurate; world sense keeps it useful. A camera fixes moved and missing objects, but simple vision is easily fooled by light — so the vision node must be able to say “not sure”.

Knowledge check

Check your understanding

Knowledge check

1. Vision mainly gives a robot…

2. Encoders and potentiometers on the joints tell the robot…

3. The block is moved 4 cm and the gripper can absorb 1.5 cm of error. The blind arm…

4. A 640 × 480 colour image in OpenCV has shape…

5. A pure red pixel read with OpenCV prints as…

6. The lights dim and the seeing arm finds no red pixels. The best behaviour is…

7. Where should servo power come from when you add motors to the vision bench?

Mini assignment

Hands-on assignment

Mini assignment

Your arm’s “eyes” plan

Decide what your desk arm needs to see, and prove with numbers why a blind arm is not enough.

Deliverables

  • A table of three tasks for your arm (for example: pick a block, sort by colour, stop when a hand comes near) with the sensor you would use for each — camera, depth camera, light sensor or switch — and why.
  • Output of `blind_vs_seeing_arm.py` with one new case of your own (a different block position or light level).
  • `first_look.py` readings (B, G, R and brightness) for three objects under two different lights.
  • Optional (bench): a short note or video of `body_and_world.ino` showing body sense and world sense separately.

Do this, step by step

  1. Run python3 blind_vs_seeing_arm.py and copy the SUMMARY of both arms into your notebook.
  2. Add one case of your own (move the block or change the light) and run again; note which arm fails.
  3. Make a three-row table: task, sensor (camera, depth camera, light sensor or switch) and one reason.
  4. Run first_look.py and write the centre B, G, R of three objects under the room light, then under a desk lamp.
  5. Optional bench: run body_and_world.ino and write one line on what the pot (body sense) and the photocell (world sense) each tell you.

Stuck? Hints, one at a time

NoteTry the task first. Read one hint, try again, and only then read the next one.
  1. The blind arm goes to REMEMBERED; if the block is farther away than GRIP_TOL_CM, it misses.
  2. OpenCV prints colours as B, G, R — a red object has a big last number.
  3. A pick task needs position, so a camera; “a hand is near” can be a simple light or distance sensor.

Challenges (pick by level)

  • 🟢 Beginner: Fix the half-pixel error: use (us.mean() + 0.5) * CM_PER_PX. What error do you get now?
  • 🟡 Intermediate: Add a second, blue block and make the seeing arm pick the one closest to the arm’s base at (0, 24) cm.
  • 🟠 Advanced: Make Case 5 work: instead of a fixed R > 150, call it red when R is much larger than G and B (for example R > 1.8 × G and R > 1.8 × B). Test light = 0.4 and 1.2.
  • 🔴 Expert: In first_look.py, find the red pixels in each camera frame with NumPy and draw a circle at their centre. Print the centre in pixels 5 times a second.
  • ⚫ Research: Read about event cameras or thermal cameras and write half a page on which robot jobs they handle better than a normal camera, with sources.

Next step

You know why a robot needs vision. In Lesson 2 — How Cameras See the World you will learn how a camera turns light from a 3D table into a 2D grid of pixels: lens, sensor, exposure, field of view and the pinhole model.

Sources

Look back: Stage 10 L16: Capstone — Build a ROS 2 6DOF Robot System. Next: Lesson 2: How Cameras See the World. Related: Stage 10 — ROS 2, Stage 4 — Electronics.

Self-attested for now — keep artifacts in your lab notebook.

Complete

Today's goal

0 of 1 done today

0-day streak
Daily goal:lessons, quizzes or assignments a day

Mark this lesson complete when you have worked the experiment, knowledge check, and assignment (even roughly).

Stuck somewhere?

Ask a doubt about this lesson

Saved with this lesson's id and your email so we can reply. Privacy · or email contact@aihubstation.in