The vision layer sends a carefully normalized, smooth 0-to-1 signal. The simulated finger turns it into two states: open and closed. This is the most important thing to understand about the twin — and every number below was measured on the model in the repository. The measurement # The same flexion was applied to all five motors, the model simulated for 3 s from rest, and the steady-state joint angles recorded — 41 flexion values from 0 to 1.
Flexion Force Index MCP / PIP / DIP Thumb CMC / MP / IP 0.000 +50.0 N −3.3° / −3.2° / −3.3° −2.9° / −2.7° / −2.7° 0.250 +25.0 N −1.7° / −1.6° / −1.7° −1.5° / −1.3° / −1.4° 0.500 0.0 N 0.0° / 0.0° / 0.0° 0.0° / 0.0° / 0.0° 0.625 −12.5 N 91.1° / 91.0° / 91.0° 90.6° / 53.9° / 45.6° 1.000 −50.0 N 94.0° / 93.8° / 93.9° 92.3° / 54.2° / 46.0° Joint limits are 90°; values just above it are MuJoCo’s soft limit being pressed.
One vision container turns webcam frames into five numbers. One simulation container turns those numbers into tendon forces. Everything else in this series is detail inside one of those two boxes — or the pipe between them. Left to right, three layers in one frame: vision (landmarks), simulation (the twin), actuation (live motor forces from MuJoCo’s Control panel). End to end # flowchart LR subgraph VISION["🐳 vision_tracker container"] direction TB A["Webcam frame
640×480 BGR"] --> B["MediaPipe Hands
21 landmarks"] B --> C["3 knuckle angles / finger
dot product"] C --> D["mean → 1 curl angle
(underactuation)"] D --> E["normalize + clip
flexion 0..1"] end subgraph TWIN["🐳 mujoco_twin container"] direction TB F["lerp
+50 N … −50 N"] --> G["data.ctrl on
pull_{finger} motor"] G --> H["spatial tendon
through 6 sites"] H --> I["3 passive hinge joints
curl"] I --> J["MuJoCo viewer"] end E -- "ROS 2 · /hand/target_flexions
sensor_msgs/JointState" --> F Stage What comes out Deep dive MediaPipe Hands 21 (x, y, z) landmarks per frame Part 4 Triplet angles 3 interior angles per finger, in radians Part 5 Averaging 1 curl angle per finger Part 6 Normalization flexion 0..1 (the thumb has its own window) Parts 7–8 ROS 2 topic JointState: names are fingers, positions are flexions Part 15 Lerp force in newtons per tendon Part 10 Tendon physics joint angles Part 11 Why the pipe carries flexions, not angles or forces # The contract between the containers is five unitless numbers: 0.0 is an open finger, 1.0 is a closed one. That choice is the architecture.
The system works, and it teaches well. Getting it to drive real servos safely is a sequence of well-scoped upgrades — each one grounded in a limitation measured earlier in this series. Where it stands # Area Today Production target Hand tracking image-normalized landmarks, one global calibration metric world landmarks, per-finger calibration, a temporal filter Command mapping flexion → ±50 N; the twin is a switch flexion → tendon length; proportional curl Physics model force motors, decorative horns, no self-contact position servos on horns, tuned stiffness, contacts for grasping Middleware one topic, default QoS, open on the LAN parameters, explicit QoS, a watchdog, SROS 2 Containers privileged, host namespaces, root, xhost least privilege, non-root, optional headless Code classes copied into three files, no tests one shared package, tests, CI Hardware simulation only a servo driver on the same topic Stage 1 · Correctness Measure the right thing, command the right quantity World landmarks. Image-normalized coordinates bend angles by up to 16° with hand orientation (Part 4). Read multi_hand_world_landmarks instead, then recalibrate.
Everything in this series in one read: how a $20 webcam ends up driving a 15-DOF CAD model in real time, why every step is deliberately explicit rather than learned, and what broke along the way. You hold your hand up to a laptop camera. On the other half of the screen, a robotic hand — designed in CAD, never manufactured — closes its fingers at the same moment yours do.
There is no glove, no marker, no depth sensor. Just an RGB webcam, two small neural networks, about forty lines of vector geometry, and a middleware stack that thinks it is talking to a real robot.
All of it is open source under AGPL-3.0 and archived with a DOI: 10.5281/zenodo.22658556.
flowchart LR A["📷 Webcam
/dev/video0"] --> B["BlazePalm
palm detector"] B --> C["Landmark regressor
21 × (x, y, z)"] C --> D["Dot-product geometry
15 interior angles"] D --> E["Normalize → flexion
0.0 straight · 1.0 curled"] E --> F["Lerp onto the URDF's
mechanical limits"] F --> G["/joint_states"] G --> H["robot_state_publisher
→ /tf"] H --> I["🖥️ RViz digital twin"] The rule that shaped the build # There is an easier version of this project. Collect a few thousand frames of a hand next to the corresponding CAD poses, train a network to map one to the other, and let gradient descent work out the relationship.
