Digital-Twin on Mulham Fetna

From tutorial project to production robot: a roadmap for the tendon hand twin

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

The system works, and it teaches well. Getting it to drive real servos safely is a sequence of well-scoped upgrades — each one grounded in a limitation measured earlier in this series.

Where it stands
#

Area	Today	Production target
Hand tracking	image-normalized landmarks, one global calibration	metric world landmarks, per-finger calibration, a temporal filter
Command mapping	flexion → ±50 N; the twin is a switch	flexion → tendon length; proportional curl
Physics model	force motors, decorative horns, no self-contact	position servos on horns, tuned stiffness, contacts for grasping
Middleware	one topic, default QoS, open on the LAN	parameters, explicit QoS, a watchdog, SROS 2
Containers	privileged, host namespaces, root, `xhost`	least privilege, non-root, optional headless
Code	classes copied into three files, no tests	one shared package, tests, CI
Hardware	simulation only	a servo driver on the same topic

Stage 1 · Correctness

Measure the right thing, command the right quantity

World landmarks. Image-normalized coordinates bend angles by up to 16° with hand orientation (Part 4). Read multi_hand_world_landmarks instead, then recalibrate.

How a webcam moves a simulated tendon-driven hand

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

One vision container turns webcam frames into five numbers. One simulation container turns those numbers into tendon forces. Everything else in this series is detail inside one of those two boxes — or the pipe between them.

Left to right, three layers in one frame: vision (landmarks), simulation (the twin), actuation (live motor forces from MuJoCo’s Control panel).

End to end
#

flowchart LR
    subgraph VISION["🐳 vision_tracker container"]
        direction TB
        A["Webcam frame
640×480 BGR"] --> B["MediaPipe Hands
21 landmarks"]
        B --> C["3 knuckle angles / finger
dot product"]
        C --> D["mean → 1 curl angle
(underactuation)"]
        D --> E["normalize + clip
flexion 0..1"]
    end
    subgraph TWIN["🐳 mujoco_twin container"]
        direction TB
        F["lerp
+50 N … −50 N"] --> G["data.ctrl on
pull_{finger} motor"]
        G --> H["spatial tendon
through 6 sites"]
        H --> I["3 passive hinge joints
curl"]
        I --> J["MuJoCo viewer"]
    end
    E -- "ROS 2 · /hand/target_flexions
sensor_msgs/JointState" --> F

Stage	What comes out	Deep dive
MediaPipe Hands	21 (x, y, z) landmarks per frame	Part 4
Triplet angles	3 interior angles per finger, in radians	Part 5
Averaging	1 curl angle per finger	Part 6
Normalization	flexion 0..1 (the thumb has its own window)	Parts 7–8
ROS 2 topic	`JointState`: names are fingers, positions are flexions	Part 15
Lerp	force in newtons per tendon	Part 10
Tendon physics	joint angles	Part 11

Why the pipe carries flexions, not angles or forces
#

The contract between the containers is five unitless numbers: 0.0 is an open finger, 1.0 is a closed one. That choice is the architecture.

Why the simulated tendon finger is a switch — and three fixes tested in MuJoCo

contact@mulhamfetna.com (Mulham Fetna) — Wed, 16 Sep 2026 00:00:00 +0000

The vision layer sends a carefully normalized, smooth 0-to-1 signal. The simulated finger turns it into two states: open and closed. This is the most important thing to understand about the twin — and every number below was measured on the model in the repository.

The measurement
#

The same flexion was applied to all five motors, the model simulated for 3 s from rest, and the steady-state joint angles recorded — 41 flexion values from 0 to 1.

Flexion	Force	Index MCP / PIP / DIP	Thumb CMC / MP / IP
0.000	+50.0 N	−3.3° / −3.2° / −3.3°	−2.9° / −2.7° / −2.7°
0.250	+25.0 N	−1.7° / −1.6° / −1.7°	−1.5° / −1.3° / −1.4°
0.500	0.0 N	0.0° / 0.0° / 0.0°	0.0° / 0.0° / 0.0°
0.625	−12.5 N	91.1° / 91.0° / 91.0°	90.6° / 53.9° / 45.6°
1.000	−50.0 N	94.0° / 93.8° / 93.9°	92.3° / 54.2° / 46.0°

Joint limits are 90°; values just above it are MuJoCo’s soft limit being pressed.

A webcam, some vector geometry, and a hand that moves

contact@mulhamfetna.com (Mulham Fetna) — Tue, 08 Sep 2026 00:00:00 +0000

Everything in this series in one read: how a $20 webcam ends up driving a 15-DOF CAD model in real time, why every step is deliberately explicit rather than learned, and what broke along the way.

You hold your hand up to a laptop camera. On the other half of the screen, a robotic hand — designed in CAD, never manufactured — closes its fingers at the same moment yours do.

There is no glove, no marker, no depth sensor. Just an RGB webcam, two small neural networks, about forty lines of vector geometry, and a middleware stack that thinks it is talking to a real robot.

All of it is open source under AGPL-3.0 and archived with a DOI: 10.5281/zenodo.22658556.

flowchart LR
    A["📷 Webcam
/dev/video0"] --> B["BlazePalm
palm detector"]
    B --> C["Landmark regressor
21 × (x, y, z)"]
    C --> D["Dot-product geometry
15 interior angles"]
    D --> E["Normalize → flexion
0.0 straight · 1.0 curled"]
    E --> F["Lerp onto the URDF's
mechanical limits"]
    F --> G["/joint_states"]
    G --> H["robot_state_publisher
→ /tf"]
    H --> I["🖥️ RViz digital twin"]

The rule that shaped the build
#

There is an easier version of this project. Collect a few thousand frames of a hand next to the corresponding CAD poses, train a network to map one to the other, and let gradient descent work out the relationship.

Digital-Twin on Mulham Fetna

From tutorial project to production robot: a roadmap for the tendon hand twin

Where it stands #

Stage 1 · Correctness

Measure the right thing, command the right quantity

How a webcam moves a simulated tendon-driven hand

End to end #

Why the pipe carries flexions, not angles or forces #

Why the simulated tendon finger is a switch — and three fixes tested in MuJoCo

The measurement #

A webcam, some vector geometry, and a hand that moves

The rule that shaped the build #

Where it stands
#

End to end
#

Why the pipe carries flexions, not angles or forces
#

The measurement
#

The rule that shaped the build
#