Mediapipe on Mulham Fetna

Calibrating hand tracking to your own hand — with real data from a live session

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

The shipped thresholds were measured on one hand with one webcam. Here is how to measure yours — and what a live recording revealed about how far off “good enough” can be while the twin still looks perfect.

Symptoms that call for calibration
#

What you see in the published flexions	Cause	Change
A fist, but flexion stays below 1.0	your fist angle is above the curled limit	raise `*_CURLED_ANGLE` to your fist reading
Flexion hits 1.0 with the hand half closed	curled limit too high	lower `*_CURLED_ANGLE`
Flexion above 0 with a relaxed open hand	your open angle is below the straight limit	lower `*_STRAIGHT_ANGLE` to your open reading
The thumb flickers	thumb window too narrow for your jitter	widen it (Part 8)

Calibrate against the numbers, not the render. With the current force-control tuning, the simulated finger snaps shut past flexion ≈ 0.51 (Part 11), so most calibration errors are invisible in the viewer. Watch ros2 topic echo /hand/target_flexions.

Real data: what a live session produced
#

MuJoCo’s viewer has a Control panel showing the live force of every motor. Since the force is \(F = 50 - 100 \cdot \text{flexion}\), every recorded frame gives back the exact flexion the tracker published: flexion = (50 − F) / 100.

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Twenty-one points per frame, on a laptop CPU, from a flat RGB image. That is what MediaPipe hands to a robot. It is easy to treat as a black box — until you compute angles from it and discover the box stretched your coordinate space.

Where MediaPipe sits
#

flowchart TB
    CAM["USB webcam
640×480 @ 30 fps · YUYV"] -->|"BGR frame"| RGB["cv2.cvtColor
BGR → RGB"]
    RGB --> MP["MediaPipe Hands
palm detector + landmark model"]
    MP -->|"21 × (x, y, z)"| ANG["Triplet angles
Part 5"]
    ANG --> AVG["Average per finger
Part 6"]
    AVG --> NORM["Flexion 0..1
Parts 7–8"]
    NORM -->|"/hand/target_flexions"| TWIN["MuJoCo twin"]

The vision layer knows nothing about MuJoCo. Its entire output is five numbers between 0.0 (open) and 1.0 (closed).

Two networks, not one
#

MediaPipe Hands is a cascade:

BlazePalm, a single-shot detector, finds a palm bounding box in the full frame. Palms, not hands: a palm is close to a rigid square; a hand with moving fingers is not.
A landmark model crops that region and regresses 21 keypoints, a hand-presence score and handedness.

In video mode (static_image_mode=False) the detector barely runs. Landmarks from frame t define the crop for frame t+1, and the detector wakes only when tracking confidence drops. That shortcut is why this pipeline holds 30 fps on a CPU.

Why the thumb needs its own thresholds in hand tracking

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Squeeze your thumb across your palm as hard as you can and, by the finger thresholds, it is only half closed. The thumb isn’t a finger with a shorter bone — it’s a different joint.

Thumb curled, fingers open. Only pull_thumb goes negative (−19.5 N, flexion 0.69); every other motor pushes open.

Hinges versus a saddle
#

Index to pinky are chains of hinge joints. A fist rolls them into a tight spiral, each knuckle approaching 90°.
The thumb hangs from the carpometacarpal (CMC) saddle joint at the wrist. It sweeps across the palm — opposition — instead of simply curling. Much of “closing the thumb” is rotation of the whole thumb, not bending at its knuckles.

What MediaPipe measures because of it
#

Interior angles come from 3D landmark positions, so the sweep only partly shows up as knuckle bend:

	Open	Fully closed	Travel
Finger — mean of MCP/PIP/DIP	≈ 3.10 rad	≈ 1.60 rad	1.50 rad
Thumb — mean of CMC/MCP/IP	≈ 2.90 rad	≈ 2.30 rad	0.60 rad

A fully closed thumb stops near 2.30 rad (~132°). Even its open angle sits below a finger’s — a relaxed thumb is never in line with the wrist.

How MediaPipe sees a hand — and exactly where it fails

contact@mulhamfetna.com (Mulham Fetna) — Tue, 08 Sep 2026 00:00:00 +0000

Twenty-one points, thirty frames a second, on a CPU, from a flat RGB image with no depth information. This is how that is possible — and the four failure modes you will meet the first time you rely on it.

Every vision-driven robotics project has a moment where the camera stops being a camera and starts being a sensor. For this one, that moment is MediaPipe Hands: a webcam frame goes in, and 21 numbered points in space come out.

It is easy to treat that as a black box. It is also a mistake, because the box has a specific shape, and its failure modes follow directly from how it was built.

It is two networks, not one
#

The single most useful thing to know about MediaPipe Hands is that it is a cascade: a detector and a regressor, with completely different jobs.

flowchart TB
    A["📷 Full frame
e.g. 640 × 480"] --> B["Stage 1 — BlazePalm
SSD detector"]
    B --> C["Oriented palm crop
256 × 256"]
    C --> D["Stage 2 — Landmark regressor
MobileNetV2-style encoder"]
    D --> E["63 floats
21 landmarks × (x, y, z)"]
    D --> F["Presence score"]
    D --> G["Handedness
left / right"]
    F -->|"confidence ≥ 0.5"| C
    F -->|"confidence < 0.5"| B

The regressor’s output, drawn back onto the frame: 21 points and the connections between them.

Stage 1 — BlazePalm detects palms, never fingers
#

This is the design decision the whole system rests on.

Mediapipe on Mulham Fetna

Calibrating hand tracking to your own hand — with real data from a live session

Symptoms that call for calibration #

Real data: what a live session produced #

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

Where MediaPipe sits #

Two networks, not one #

Why the thumb needs its own thresholds in hand tracking

Hinges versus a saddle #

What MediaPipe measures because of it #

How MediaPipe sees a hand — and exactly where it fails

It is two networks, not one #

Stage 1 — BlazePalm detects palms, never fingers #

Symptoms that call for calibration
#

Real data: what a live session produced
#

Where MediaPipe sits
#

Two networks, not one
#

Hinges versus a saddle
#

What MediaPipe measures because of it
#

It is two networks, not one
#

Stage 1 — BlazePalm detects palms, never fingers
#