Docker

The single-script version felt smoother than the containers. The obvious suspect was Docker. The measurements say otherwise — and point at something more interesting: two programs sharing one integrated GPU. Setup # Laptop with an Intel Comet Lake CPU (12 threads) and Intel UHD Graphics; Linux on Wayland with XWayland; USB webcam at 640×480, 30 fps, YUYV. Each row: mean of 150 frames after a 10-frame warm-up, no hand in view. Measured 2026-09-16. The result # Condition Camera read MediaPipe imshow Loop Vision alone, container 10.0 ms 19.0 ms 4.5 ms 29.8 Hz Vision alone, host venv (MediaPipe 0.10.11) 10.6 ms 19.2 ms 3.9 ms 29.7 Hz Container vision + 1-core CPU burner 8.2 ms 23.5 ms 3.9 ms 28.1 Hz Host vision + plain MuJoCo viewer on host (no Docker, no ROS) 1.6 ms 54.9 ms 5.9 ms 16.0 Hz Container vision + mujoco_twin capped at 30 Hz 1.7 ms 63.7 ms 8.1 ms 13.6 Hz Container vision + mujoco_twin as shipped 1.9 ms 85.6 ms 10.2 ms 10.2 Hz End to end, ros2 topic hz /hand/target_flexions read 30.0 Hz with only vision_tracker running and 10.7 Hz once mujoco_twin started.

Every entry here was hit, or deliberately checked, while building this project. Error messages are quoted exactly so a search for the message lands on the fix. Camera and windows # RuntimeError: Failed to open camera at index 0 # Something else holds the webcam — the standalone script, a previous container, a browser tab. Run docker compose down and close video apps. A camera opens in one process at a time. The camera is another node. ls /dev/video*, then map that device and set CAMERA_INDEX (many webcams expose /dev/video0 for frames and /dev/video1 for metadata — use the first). The device isn’t mapped. Check with docker compose config | grep video. Windows don’t open — cannot connect to X server, could not connect to display # Run ./setup_host.sh (it runs xhost +local:root). The permission resets when you log out. Make sure echo $DISPLAY on the host prints :0, or export DISPLAY before docker compose up. On Wayland, confirm XWayland is running: ls /tmp/.X11-unix/ should list X0. Black, blank or garbled OpenCV window # QT_X11_NO_MITSHM=1 must reach the container. It’s in the shared Compose environment — if a service defines its own environment:, it must merge the shared anchor with <<: *ros-env rather than replace it (Part 16).

Clone, run one setup script, run one Compose command — and a webcam window and a MuJoCo viewer open side by side, with a simulated hand that closes when you close yours. Here are four ways in, from the full stack down to the bare model. The goal of this article: both windows open, and the twin follows your hand. What you need # Requirement Check with Notes Linux desktop, X11 or XWayland echo $DISPLAY → :0 Wayland sessions work through XWayland Docker Engine + Compose v2 docker compose version Built with Docker 29.8 / Compose 5.5 A webcam ls /dev/video0 Close any other app using it GPU device nodes ls /dev/dri Intel/AMD out of the box; NVIDIA needs the container toolkit (Part 18) ~5 GB of disk images: vision 2.9 GB, twin 1.7 GB No ROS installation is needed on the host — ROS 2 Jazzy lives inside the containers. Four ways to run it # Docker Compose (full stack) Single Python script Twin only, no camera Model viewer git clone https://github.com/mulhamfetna/ros2-tendon-driven-hand-mujoco-digital-twin-vision-teleoperation.git cd ros2-tendon-driven-hand-mujoco-digital-twin-vision-teleoperation ./setup_host.sh # X11 access for the containers + camera/GPU checks (once per login) docker compose up --build # builds both images, starts vision_tracker and mujoco_twin Stop with Ctrl+C, then docker compose down to release the camera.

“The topic shows up in ros2 topic list, but echo prints nothing” — the classic ROS-in-Docker symptom. Here’s why it happens, how two containers in this project share a network and a block of memory instead, and what that costs. How ROS 2 nodes find each other # ROS 2 has no master. Nodes discover each other through DDS — in Jazzy, eProsima Fast DDS by default — using the SPDP protocol: Each participant announces itself over UDP multicast (239.255.0.1) on ports derived from the domain ID \(d\): discovery multicast on \(7400 + 250d\), unicast on \(7410 + 250d + 2p\) for participant \(p\). For domain 42: UDP 17900 and 17910+. Peers exchange their topic endpoints; matching publishers and subscribers connect. Data flows over the best transport both support — shared memory when they share a host and /dev/shm, UDP otherwise. Why Docker’s default network breaks it # Compose attaches services to a bridge network — a private NATed subnet. Multicast isn’t reliably routed across it, and machines on your LAN can’t reach container IPs at all.

Containers are headless by design. This project needs two windows, a webcam and a GPU — so the Compose file spends most of its lines punching carefully chosen holes back through the isolation. Two containers, two windows, one X display: the Qt/OpenCV tracker (vision_tracker) and the GLFW MuJoCo viewer (mujoco_twin), recorded on Wayland through XWayland. Windows: pass the X11 socket through # environment: DISPLAY: ${DISPLAY:-:0} QT_X11_NO_MITSHM: 1 volumes: - /tmp/.X11-unix:/tmp/.X11-unix:rw Piece What it does /tmp/.X11-unix mount the X server listens on a Unix socket here (X0 for :0); mounting it gives the container a line to it DISPLAY tells X clients (GLFW, Qt) which display to use; defaults to :0 QT_X11_NO_MITSHM=1 stops Qt — OpenCV’s imshow backend — from using MIT-SHM, which fails across containers and gives blank or garbled windows xhost +local:root (in setup_host.sh) the X server refuses untrusted clients; this admits local root, which is who the containers run as On Wayland # Wayland sessions (KDE Plasma, GNOME) still run XWayland on :0, and both windows open through it — the recording above was made exactly that way. Running MuJoCo natively on a Wayland host, GLFW may warn Wayland: The platform does not provide the window position; it’s harmless.

Both images hold dependencies and nothing else. The code and the robot model are mounted from your checkout at runtime — so an edit is a five-second restart, not a five-minute rebuild. The big picture # flowchart TB subgraph HOST["🐧 Linux host"] CAMDEV["/dev/video0"] GPU["/dev/dri · Intel iGPU"] X11["/tmp/.X11-unix XWayland :0"] SHM["/dev/shm Fast DDS segments"] NET["host network UDP multicast · domain 42"] REPO["repository checkout"] subgraph VT["🐳 vision_tracker · 2.9 GB image"] VN["vision_tracker_node.py mediapipe 0.10.14 · OpenCV"] end subgraph MT["🐳 mujoco_twin · 1.7 GB image"] MN["mujoco_twin_node.py mujoco 3.13.0 · GLFW"] end end CAMDEV --> VN GPU --> VN GPU --> MN X11 <--> VN X11 <--> MN VN <--> SHM <--> MN VN <--> NET <--> MN REPO -. "bind mount .:/workspace:ro" .-> VN REPO -. "bind mount .:/workspace:ro" .-> MN Repository layout # . ├── docker-compose.yml # both services, shared namespaces ├── setup_host.sh # xhost + device checks, once per login ├── vision_tracker/ │ ├── Dockerfile # ros:jazzy + mediapipe==0.10.14 │ ├── .dockerignore # src/ is mounted, so keep it out of the build context │ └── src/vision_tracker_node.py ├── mujoco_twin/ │ ├── Dockerfile # ros:jazzy + mujoco==3.13.0 │ ├── .dockerignore # src/ and model/ are mounted │ ├── src/mujoco_twin_node.py │ └── model/ # scene.xml → robot.xml (+ tendons.xml), assets/, config.json ├── standalone/main.py # the same pipeline, one process └── docs/ Each service folder is its own build context: editing the vision Dockerfile never invalidates the twin’s image cache, and neither build uploads the 13 MB of meshes it doesn’t need.

Web developers containerize to isolate processes. Roboticists containerize and then spend the rest of the day punching holes back through the isolation — for the camera, the GPU, the display server and the network. ROS 2 Jazzy is hard-locked to Ubuntu 24.04. That single fact is reason enough to containerize: without it, adopting a ROS distribution means adopting an operating system version, on every machine that will ever run the code. But a ROS container is a strange artifact. A web service container wants isolation — that is the product. A robotics container needs a webcam, a GPU, a display server and multicast networking, all of which live outside it. You end up building a box and then carefully cutting four holes in it. Here is every hole in this project’s compose file and what it is for. Cold start to live tracking. Note hand_msgs being compiled separately inside two different containers — that duplication is deliberate and load-bearing. flowchart TB subgraph HOST["🖥️ Host — Ubuntu / Kubuntu"] X11["X11 socket /tmp/.X11-unix"] CAM["/dev/video0"] GPU["/dev/dri"] NET["host network UDP multicast"] end subgraph C["Active containers · ROS_DOMAIN_ID=42"] HT["hand_tracker"] RV["ros_rviz"] TS["topic_sniffer"] end GZ["gazebo_sim (commented out)"]:::parked classDef parked stroke-dasharray: 5 5,opacity:0.55 CAM --> HT GPU --> HT GPU --> RV X11 --> HT X11 --> RV NET <--> HT NET <--> RV NET <--> TS Hole 1 — the network wall has to come down # network_mode: "host" ipc: host pid: host This is the one that breaks stacks silently, so it is worth understanding rather than copying.