hand-tracking-drawing

Hand Tracking Drawing

Draw in the air with your hand, then view the drawing as a rotatable 3D object.

DOCKER IMAGE HERE

Model

Uses the MediaPipe Hand Landmarker (models/hand_landmarker.task), which returns 21 keypoints per hand. Gestures are classified from ratios and joint angles between those keypoints, so recognition is independent of hand size, distance, or handedness.

Logic

• One index finger - draw
• Index + middle - move cursor without drawing
• Fist (hold) - grab the drawing; move the fist to rotate it in 3D
• Two fists - scale (canvas in 2D, object in 3D)
• Open palm - release, back to drawing
• OK (hold) - clear the canvas
• Pinky (hold) - undo the last stroke
• Middle finger - shows :(( for a second
• Both palms waving - say goodbye, closes the app

Keys: S save, C clear, U undo, 1..6 color, [ / ] thickness, SPACE 2D/3D, R reset view, K skeleton, D debug, H help, Q quit.

3D Drawing

The 3D Drawing mode uses the YOLO depth model (models/yolo26s-depth.pt), which returns a per pixel depth map in meters. Every drawn point takes its Z from the metric depth under the fingertip, so moving the hand closer or further really draws into the scene: the stroke is a 3D curve in space, not a flat picture placed on a board. The first drawn point sets the zero plane, everything after that is relative to it. Depth runs in a background thread, so tracking and drawing stay at camera speed.

Raw monocular depth flickers, so the pen does not follow it directly: the global scale drift of the map is cancelled against the static background, the fingertip depth goes through a one euro filter with a noise deadband and a speed limit, and the newest points of a stroke are low passed along the curve. A circle drawn without moving the hand comes out flat, while a deliberate push of 25 cm is kept in full.

By default strokes are colored by a depth gradient - warm near the camera, cool further away. 1..6 switch to a solid color, G goes back to the gradient.

Gestures and keys are the same as in 2D, with two additions:

• Two ring fingers (hold) - toggle depth colors for the whole frame, drawing keeps working
• Open palm (hold) - reset the view

In 3D the fist rotates the drawing itself and two fists scale it, SPACE toggles depth colors, R resets the view.

Run

pip install -r requirements.txt
python src/main.py

Pick the mode in the menu, or skip it:

python src/main.py --no-menu --mode 3d
python src/main.py --no-menu --mode 3d --depth-size 448 --depth-every 1

by Salimli Ayzek (Салимли Айзек): https://mathematiclove.github.io