Computer-Vision on Mulham Fetna

Calibrating hand tracking to your own hand — with real data from a live session

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

The shipped thresholds were measured on one hand with one webcam. Here is how to measure yours — and what a live recording revealed about how far off “good enough” can be while the twin still looks perfect.

Symptoms that call for calibration
#

What you see in the published flexions	Cause	Change
A fist, but flexion stays below 1.0	your fist angle is above the curled limit	raise `*_CURLED_ANGLE` to your fist reading
Flexion hits 1.0 with the hand half closed	curled limit too high	lower `*_CURLED_ANGLE`
Flexion above 0 with a relaxed open hand	your open angle is below the straight limit	lower `*_STRAIGHT_ANGLE` to your open reading
The thumb flickers	thumb window too narrow for your jitter	widen it (Part 8)

Calibrate against the numbers, not the render. With the current force-control tuning, the simulated finger snaps shut past flexion ≈ 0.51 (Part 11), so most calibration errors are invisible in the viewer. Watch ros2 topic echo /hand/target_flexions.

Real data: what a live session produced
#

MuJoCo’s viewer has a Control panel showing the live force of every motor. Since the force is $F = 50 - 100 \cdot \text{flexion}$, every recorded frame gives back the exact flexion the tracker published: flexion = (50 − F) / 100.

Finger joint angles from three hand landmarks: the dot-product geometry

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Every knuckle angle in this project comes from three landmarks and one dot product. No learning, no lookup table — just the definition of the angle between two vectors, plus two guards that keep a single glitchy frame from sending NaN into a motor command.

Three points make an angle
#

An angle needs a vertex and two rays. A knuckle is the vertex; the two bones meeting there are the rays:

a base point, where the previous bone starts,
the vertex — the knuckle being measured,
an end point, where the next bone ends.

flowchart LR
    P1(("p1
base")) -- "v1 = p1 − p2" --- P2(("p2
vertex
knuckle"))
    P2 -- "v2 = p3 − p2" --- P3(("p3
end"))

The triplet table
#

self.finger_triplets = {
    "thumb":  [(0, 1, 2), (1, 2, 3), (2, 3, 4)],
    "index":  [(0, 5, 6), (5, 6, 7), (6, 7, 8)],
    "middle": [(0, 9, 10), (9, 10, 11), (10, 11, 12)],
    "ring":   [(0, 13, 14), (13, 14, 15), (14, 15, 16)],
    "pinky":  [(0, 17, 18), (17, 18, 19), (18, 19, 20)]
}

Landmark 0 — the wrist — starts every finger’s first triplet.

Finger	Triplet	Vertex	Joint measured
Index	(0, 5, 6)	5	MCP — joins finger to palm
Index	(5, 6, 7)	6	PIP — middle knuckle
Index	(6, 7, 8)	7	DIP — fingertip knuckle
Thumb	(0, 1, 2)	1	CMC — saddle joint at the wrist
Thumb	(1, 2, 3)	2	MCP
Thumb	(2, 3, 4)	3	IP

Why every first triplet starts at 0. The palm has no landmark of its own, so the wrist → knuckle line stands in for the metacarpal bone. It isn’t exactly collinear with a straight finger — ring and pinky metacarpals fan outward — so a relaxed straight finger rarely measures a full $\pi$. That’s part of why the calibrated “straight” threshold is 3.10 rad rather than 3.14.

How a webcam moves a simulated tendon-driven hand

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

One vision container turns webcam frames into five numbers. One simulation container turns those numbers into tendon forces. Everything else in this series is detail inside one of those two boxes — or the pipe between them.

Left to right, three layers in one frame: vision (landmarks), simulation (the twin), actuation (live motor forces from MuJoCo’s Control panel).

End to end
#

flowchart LR
    subgraph VISION["🐳 vision_tracker container"]
        direction TB
        A["Webcam frame
640×480 BGR"] --> B["MediaPipe Hands
21 landmarks"]
        B --> C["3 knuckle angles / finger
dot product"]
        C --> D["mean → 1 curl angle
(underactuation)"]
        D --> E["normalize + clip
flexion 0..1"]
    end
    subgraph TWIN["🐳 mujoco_twin container"]
        direction TB
        F["lerp
+50 N … −50 N"] --> G["data.ctrl on
pull_{finger} motor"]
        G --> H["spatial tendon
through 6 sites"]
        H --> I["3 passive hinge joints
curl"]
        I --> J["MuJoCo viewer"]
    end
    E -- "ROS 2 · /hand/target_flexions
sensor_msgs/JointState" --> F

Stage	What comes out	Deep dive
MediaPipe Hands	21 (x, y, z) landmarks per frame	Part 4
Triplet angles	3 interior angles per finger, in radians	Part 5
Averaging	1 curl angle per finger	Part 6
Normalization	flexion 0..1 (the thumb has its own window)	Parts 7–8
ROS 2 topic	`JointState`: names are fingers, positions are flexions	Part 15
Lerp	force in newtons per tendon	Part 10
Tendon physics	joint angles	Part 11

Why the pipe carries flexions, not angles or forces
#

The contract between the containers is five unitless numbers: 0.0 is an open finger, 1.0 is a closed one. That choice is the architecture.

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Twenty-one points per frame, on a laptop CPU, from a flat RGB image. That is what MediaPipe hands to a robot. It is easy to treat as a black box — until you compute angles from it and discover the box stretched your coordinate space.

Where MediaPipe sits
#

flowchart TB
    CAM["USB webcam
640×480 @ 30 fps · YUYV"] -->|"BGR frame"| RGB["cv2.cvtColor
BGR → RGB"]
    RGB --> MP["MediaPipe Hands
palm detector + landmark model"]
    MP -->|"21 × (x, y, z)"| ANG["Triplet angles
Part 5"]
    ANG --> AVG["Average per finger
Part 6"]
    AVG --> NORM["Flexion 0..1
Parts 7–8"]
    NORM -->|"/hand/target_flexions"| TWIN["MuJoCo twin"]

The vision layer knows nothing about MuJoCo. Its entire output is five numbers between 0.0 (open) and 1.0 (closed).

Two networks, not one
#

MediaPipe Hands is a cascade:

BlazePalm, a single-shot detector, finds a palm bounding box in the full frame. Palms, not hands: a palm is close to a rigid square; a hand with moving fingers is not.
A landmark model crops that region and regresses 21 keypoints, a hand-presence score and handedness.

In video mode (static_image_mode=False) the detector barely runs. Landmarks from frame t define the crop for frame t+1, and the detector wakes only when tracking confidence drops. That shortcut is why this pipeline holds 30 fps on a CPU.

Normalizing finger curl: from radians to a 0–1 flexion

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Radians belong to the camera. Newtons belong to the robot. The number that crosses between them is a plain percentage — and producing it takes one inverted formula and one clip.

The formula
#

RAW_STRAIGHT_ANGLE = 3.10  # ~177.6° — open finger
RAW_CURLED_ANGLE   = 1.60  # ~91.7° — fully curled finger

flexion = (straight_limit - avg_angle) / (straight_limit - curled_limit)
flexions[finger_name] = float(np.clip(flexion, 0.0, 1.0))

$$\text{flexion} = \operatorname{clip}\!\left(\frac{\theta_\text{straight} - \bar\theta}{\theta_\text{straight} - \theta_\text{curled}},\; 0,\; 1\right)$$

The three pieces
#

Denominator — the range. $3.10 - 1.60 = 1.50$ rad of travel. It sets the scale.

Underactuation: why three knuckle angles become one tendon command

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

avg_angle = np.mean(angles) looks like noise filtering. It isn’t. It is a mechanical decision: a human finger has three joints you can move separately, and this robot finger has one string.

Not smoothing — compression
#

“Averaging” in a sensor pipeline usually means averaging over time to reduce noise. Nothing here keeps history between frames. The mean is taken across space — over the three joints of one finger in a single frame — to solve a problem called underactuation.

The 3-to-1 problem
#

	Human finger	Robot finger
Joints	3 (MCP, PIP, DIP)	3 hinges (`_mcp`, `_pip`, `*_dip`)
Independent actuators	many muscles; joints move semi-independently	1 flexor tendon, 1 motor
Degrees of freedom you can command	~3	1

A system with fewer actuators than joints is underactuated. A single tendon threads all three joints of each robot finger, so the only command is “pull this string with force F” — and the joints share that pull according to routing geometry and dynamics.

One string, six via-points, three joints. Pull it and all three knuckles move together.

So the vision layer must compress three human measurements into one robot command.

What limits the frame rate: MediaPipe, Docker and a MuJoCo viewer on one laptop

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

The single-script version felt smoother than the containers. The obvious suspect was Docker. The measurements say otherwise — and point at something more interesting: two programs sharing one integrated GPU.

Setup
#

Laptop with an Intel Comet Lake CPU (12 threads) and Intel UHD Graphics; Linux on Wayland with XWayland; USB webcam at 640×480, 30 fps, YUYV. Each row: mean of 150 frames after a 10-frame warm-up, no hand in view. Measured 2026-09-16.

The result
#

Condition	Camera read	MediaPipe	`imshow`	Loop
Vision alone, container	10.0 ms	19.0 ms	4.5 ms	29.8 Hz
Vision alone, host venv (MediaPipe 0.10.11)	10.6 ms	19.2 ms	3.9 ms	29.7 Hz
Container vision + 1-core CPU burner	8.2 ms	23.5 ms	3.9 ms	28.1 Hz
Host vision + plain MuJoCo viewer on host (no Docker, no ROS)	1.6 ms	54.9 ms	5.9 ms	16.0 Hz
Container vision + `mujoco_twin` capped at 30 Hz	1.7 ms	63.7 ms	8.1 ms	13.6 Hz
Container vision + `mujoco_twin` as shipped	1.9 ms	85.6 ms	10.2 ms	10.2 Hz

End to end, ros2 topic hz /hand/target_flexions read 30.0 Hz with only vision_tracker running and 10.7 Hz once mujoco_twin started.

Why the thumb needs its own thresholds in hand tracking

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

Squeeze your thumb across your palm as hard as you can and, by the finger thresholds, it is only half closed. The thumb isn’t a finger with a shorter bone — it’s a different joint.

Thumb curled, fingers open. Only pull_thumb goes negative (−19.5 N, flexion 0.69); every other motor pushes open.

Hinges versus a saddle
#

Index to pinky are chains of hinge joints. A fist rolls them into a tight spiral, each knuckle approaching 90°.
The thumb hangs from the carpometacarpal (CMC) saddle joint at the wrist. It sweeps across the palm — opposition — instead of simply curling. Much of “closing the thumb” is rotation of the whole thumb, not bending at its knuckles.

What MediaPipe measures because of it
#

Interior angles come from 3D landmark positions, so the sweep only partly shows up as knuckle bend:

	Open	Fully closed	Travel
Finger — mean of MCP/PIP/DIP	≈ 3.10 rad	≈ 1.60 rad	1.50 rad
Thumb — mean of CMC/MCP/IP	≈ 2.90 rad	≈ 2.30 rad	0.60 rad

A fully closed thumb stops near 2.30 rad (~132°). Even its open angle sits below a finger’s — a relaxed thumb is never in line with the wrist.

A webcam, some vector geometry, and a hand that moves

contact@mulhamfetna.com (Mulham Fetna) — Tue, 08 Sep 2026 00:00:00 +0000

Everything in this series in one read: how a $20 webcam ends up driving a 15-DOF CAD model in real time, why every step is deliberately explicit rather than learned, and what broke along the way.

You hold your hand up to a laptop camera. On the other half of the screen, a robotic hand — designed in CAD, never manufactured — closes its fingers at the same moment yours do.

There is no glove, no marker, no depth sensor. Just an RGB webcam, two small neural networks, about forty lines of vector geometry, and a middleware stack that thinks it is talking to a real robot.

All of it is open source under AGPL-3.0 and archived with a DOI: 10.5281/zenodo.22658556.

flowchart LR
    A["📷 Webcam
/dev/video0"] --> B["BlazePalm
palm detector"]
    B --> C["Landmark regressor
21 × (x, y, z)"]
    C --> D["Dot-product geometry
15 interior angles"]
    D --> E["Normalize → flexion
0.0 straight · 1.0 curled"]
    E --> F["Lerp onto the URDF's
mechanical limits"]
    F --> G["/joint_states"]
    G --> H["robot_state_publisher
→ /tf"]
    H --> I["🖥️ RViz digital twin"]

The rule that shaped the build
#

There is an easier version of this project. Collect a few thousand frames of a hand next to the corresponding CAD poses, train a network to map one to the other, and let gradient descent work out the relationship.

How MediaPipe sees a hand — and exactly where it fails

contact@mulhamfetna.com (Mulham Fetna) — Tue, 08 Sep 2026 00:00:00 +0000

Twenty-one points, thirty frames a second, on a CPU, from a flat RGB image with no depth information. This is how that is possible — and the four failure modes you will meet the first time you rely on it.

Every vision-driven robotics project has a moment where the camera stops being a camera and starts being a sensor. For this one, that moment is MediaPipe Hands: a webcam frame goes in, and 21 numbered points in space come out.

It is easy to treat that as a black box. It is also a mistake, because the box has a specific shape, and its failure modes follow directly from how it was built.

It is two networks, not one
#

The single most useful thing to know about MediaPipe Hands is that it is a cascade: a detector and a regressor, with completely different jobs.

flowchart TB
    A["📷 Full frame
e.g. 640 × 480"] --> B["Stage 1 — BlazePalm
SSD detector"]
    B --> C["Oriented palm crop
256 × 256"]
    C --> D["Stage 2 — Landmark regressor
MobileNetV2-style encoder"]
    D --> E["63 floats
21 landmarks × (x, y, z)"]
    D --> F["Presence score"]
    D --> G["Handedness
left / right"]
    F -->|"confidence ≥ 0.5"| C
    F -->|"confidence < 0.5"| B

The regressor’s output, drawn back onto the frame: 21 points and the connections between them.

Stage 1 — BlazePalm detects palms, never fingers
#

This is the design decision the whole system rests on.

Computer-Vision on Mulham Fetna

Calibrating hand tracking to your own hand — with real data from a live session

Symptoms that call for calibration #

Real data: what a live session produced #

Finger joint angles from three hand landmarks: the dot-product geometry

Three points make an angle #

The triplet table #

How a webcam moves a simulated tendon-driven hand

End to end #

Why the pipe carries flexions, not angles or forces #

MediaPipe Hands for robotics — and the 16° angle trap in normalized landmarks

Where MediaPipe sits #

Two networks, not one #

Normalizing finger curl: from radians to a 0–1 flexion

The formula #

The three pieces #

Underactuation: why three knuckle angles become one tendon command

Not smoothing — compression #

The 3-to-1 problem #

What limits the frame rate: MediaPipe, Docker and a MuJoCo viewer on one laptop

Setup #

The result #

Why the thumb needs its own thresholds in hand tracking

Hinges versus a saddle #

What MediaPipe measures because of it #

A webcam, some vector geometry, and a hand that moves

The rule that shaped the build #

How MediaPipe sees a hand — and exactly where it fails

It is two networks, not one #

Stage 1 — BlazePalm detects palms, never fingers #

Symptoms that call for calibration
#

Real data: what a live session produced
#

Three points make an angle
#

The triplet table
#

End to end
#

Why the pipe carries flexions, not angles or forces
#

Where MediaPipe sits
#

Two networks, not one
#

The formula
#

The three pieces
#

Not smoothing — compression
#

The 3-to-1 problem
#

Setup
#

The result
#

Hinges versus a saddle
#

What MediaPipe measures because of it
#

The rule that shaped the build
#

It is two networks, not one
#

Stage 1 — BlazePalm detects palms, never fingers
#