Hand Geometry Canvas

What I wanted to solve

I did not want a demo that recognizes “thumbs up.” I wanted something more abstract: use my hands to manipulate objects — create, grab, move, resize, throw, discard — as if the objects were physically there.

And I wanted that capability to outlive any one app. The canvas on screen is one consumer. The real goal is a stream of hand events that any application can listen to and apply to its objects — a UI, an agent, a prototype, something that does not embed a camera pipeline.

So the problem splits in two:

Turn noisy landmarks into a trustworthy intention (pinch, fist / grab, move) for one or two independent hands.
Publish that intention as abstract events, without knowing what the consumer will draw.

Detecting a hand is the easy part. Making the same intention survive distance to the camera, landmark jitter, and mid-gesture noise — and then not tying that signal to rectangles — is the work.

What I built

Two pieces, on purpose — and after iteration, a clear ownership of concerns:

hand-gesture — a separate project: camera → landmarks → abstract HandEvents. It does not know about shapes, trash, or rendering. The shared gesture detector lives here: Pinch* (thumb–index), Grab* (closed fist / puño), and always-on HandMove (palm cursor). It can publish those events over a local WebSocket so any consumer can listen.
hand_canvas — the OpenCV lab that consumes those gestures and owns everything about figures: create / stretch, owner–helper resize lock, sweep-to-clear, trash, fling physics. It maps incoming events onto its pointer/grab types and adds sweep on top.

I measured and broke things in the canvas first; then pulled recognition into hand-gesture so the canvas (and anything else) only has to consume events. The canvas remains the proof that the signal is rich enough to manipulate objects.

Today the public contract is 2D: image-space positions and a small gesture vocabulary (pinch.*, grab.*, hand.move). That is enough for a UI to create, select, and move things.

How I measured — and what failed first

I did not tune by feel alone. I recorded myself pinching, opening, grabbing, and throwing; plotted the decision variables frame by frame against the raw stream; and changed one hypothesis at a time when the plot disagreed with what my hand had done.

That loop is the real method: try a rule → watch it lie on a recording → change the quantity or the structure, not just the knobs.

What I tried, what broke, and why the math showed up

1. Absolute distances → ratios (so distance to the camera stops deciding for you)

The first pinch detector used absolute floors and ceilings on the thumb–index gap in image space. Same pinch, step back from the camera: the gap shrinks, the clamps fire wrong. The plots said the hand was right and the variable was wrong.

The raw gap is just a length:

g = \lVert \mathbf{p}_{\mathrm{thumb}} - \mathbf{p}_{\mathrm{index}} \rVert

Under a uniform scale of the hand in the image ( $\mathbf{p} \mapsto \lambda\mathbf{p}$ — step closer / farther), $g$ picks up $\lambda$ . So I normalize by a hand-size measured in the same frame:

s = \operatorname{median}\Bigl( \bigl\{\lVert \mathbf{p}_i - \mathbf{p}_{\mathrm{wrist}}\rVert : i \in \mathrm{MCP}\bigr\} \cup \bigl\{\lVert \mathbf{p}_{\mathrm{index}} - \mathbf{p}_{\mathrm{pinky}}\rVert\bigr\} \Bigr)

r_{\mathrm{pinch}} = \frac{g}{s}

Both numerator and denominator scale with $\lambda$ , so $r$ stays roughly constant. Threshold a dimensionless pose measure, not a length in pixels.

Same pinch at different camera distances: raw gap tracks proximity; the ratio stays flat

After switching to $r$ , the same recordings that broke the absolute clamps started lining up. That commit was about changing what I thresholded — not adding more knobs on a bad signal.

2. One cutoff → median window + hysteresis (filter first, then decide)

Landmarks jitter. A single cutoff on $r(t)$ flickers: pinch / not / pinch. I had been treating two jobs as one knob. They are not:

Estimate the pose measure over a short window ( $N = 5$ frames) with a median, not a mean — one outlier cannot move the estimate:

\tilde{r}(t) = \operatorname{median}\bigl\{ r(t),\, r(t-1),\, \ldots,\, r(t-N+1) \bigr\}

Positions stay live so the cursor does not lag; only the scalars that feed the detector are filtered.

Decide enter / leave with hysteresis (a Schmitt trigger on $\tilde{r}$ ):

\begin{aligned} \text{enter pinch} &\quad\text{when}\quad \tilde{r} < \tau_{\mathrm{on}} = 0.32 \\ \text{leave pinch} &\quad\text{when}\quad \tilde{r} > \tau_{\mathrm{off}} = 0.55 \end{aligned}

Between $\tau_{\mathrm{on}}$ and $\tau_{\mathrm{off}}$ the previous phase holds. Filtering without hysteresis still chatters; hysteresis without filtering still trips on single-frame spikes. Order matters: filter first, then transition.

Pinch state machine with ON/OFF thresholds and a 5-frame median

Those numbers came from the plots, not from a paper. I moved $\tau_{\mathrm{on}}$ , $\tau_{\mathrm{off}}$ , and $N$ until deliberate crossings were clean and accidental ones were rare.

3. Fixed release → adaptive release (loose pinches were dying early)

A fixed $\tau_{\mathrm{off}}$ still failed a real case: a loose pinch that entered near $\tau_{\mathrm{on}}$ had almost no travel before $0.55$ , so relaxing mid-drag released and committed half-drawn figures.

The exit became adaptive to the gap I was holding — still a state-machine parameter, not another smoother:

\tau_{\mathrm{release}} = \operatorname{clamp}\bigl( r_{\mathrm{held}} \cdot \alpha,\; \tau_{\mathrm{off}},\; \tau_{\max} \bigr)

$\alpha$ went $2.2$ (too sticky) → $1.4$ once recordings showed deliberate opens fighting the band.

Experimental pinch-ratio plot from a real recording: raw vs median, release at OFF

4. When thresholds cannot help — change the interaction rule

Some failures are not threshold problems. A hard pinch aimed at the camera projects like a fist — the distributions sit on top of each other. No amount of filtering invents separability that is not in the observation.

I stopped pretending a constant could, and changed the interaction structure: over empty space a tight contact may draw; over a figure a fist grabs. Measuring told me when to quit optimizing a number and redesign the rule.

5. Two hands, one object — concurrency where the objects live

Once either hand alone could manipulate objects, the next failure was both hands writing the same figure in one frame: hit-tests stole targets, resize and move fought, the transform became nonsense.

Independent recognition per hand (separate gesture state in hand-gesture) does not solve this. It only says the signals are isolated. The conflict is downstream, at the object — so the lock lives in hand_canvas, not in the event protocol. A remote consumer that listens over WebSocket applies its own concurrency policy to its objects.

In the canvas, the model is a lock with role-specialized operations:

Role	Who	Capability on the locked resource
Owner	First hand to acquire the figure	Exclusive move. Owns the lock.
Helper	Second hand that starts an action inside the already-locked figure	Cannot steal and cannot translate. Motion remaps to resize.

Under contention, the second writer’s operation changes type so the two updates commute in intent (translate ∥ scale) instead of both writing position. That is the engineering substance behind “two-handed resize.”

Creates have the same problem on empty space: when both hands pinch in the same frame, hit-tests use a pre-batch snapshot so one create cannot steal the other’s empty target.

6. Release timing + exponential drag — so a throw feels like a throw

Grabbing is only half the story. Letting go while moving should not teleport to a stop.

Early versions confirmed an open hand over consecutive frames — more frames if the palm was moving — because a travelling arm smears and briefly looks open. That still dropped figures mid-throw. The fix was asymmetric: starting a fist stays deliberate; ending one drops as soon as fingertips sit clearly extended.

Then the object inherits the hand’s velocity $\mathbf{v}_0$ . Free flight uses exponential drag with time constant $\tau = 0.26\,\mathrm{s}$ :

\mathbf{v}(t) = \mathbf{v}_0\, e^{-t/\tau} \qquad\Rightarrow\qquad \Delta x = \int_0^{\infty} v_0\, e^{-t/\tau}\,\mathrm{d}t = v_0\,\tau

Twice the throw ≈ twice the coast. Edges bounce with restitution $e = 0.45$ so energy dies and figures settle inside the frame. Slow releases are drops, not flings.

Exponential drag: travel asymptotes to v0 tau

That physics lives in the canvas lab. The events hand-gesture emits stay abstract. A consumer that wants fling implements momentum on its side — or ignores it.

Concepts behind the choices

Problem	What I used
Same pinch fails at different camera distances	$r = g / s$ (scale-free ratio)
Gesture flickers on jitter	$\tilde{r} = \operatorname{median}(\cdot)$ then $\tau_{\mathrm{on}} \neq \tau_{\mathrm{off}}$
Loose pinch releases too early	$\tau_{\mathrm{release}} = \operatorname{clamp}(r_{\mathrm{held}}\cdot\alpha,\,\ldots)$
Pinch and fist look identical in projection	Stop tuning; redesign the interaction rule
Two hands fight over one figure	Owner / helper lock at the object layer
Throw feels dead on release	$\mathbf{v}(t)=\mathbf{v}_0 e^{-t/\tau}$ , travel $\approx v_0\tau$

`hand-gesture` as a separate project

hand-gesture is not “the canvas with the vision code extracted.” It is another project: it owns the camera, the landmarks, and the gesture vocabulary. The canvas is one consumer among others.

I did not want to ship this as a library other apps have to import, and I did not want to wire the two projects together directly (shared process, shared types, shared camera ownership). I only wanted other apps to consume the events.

So hand-gesture publishes over a localhost WebSocket. Events serialize to a stable JSON shape (type, hand_id, position, and when relevant delta_position or fingers_together) and broadcast to every connected client. Dead clients drop without taking down the camera loop. Today the schema is pinch.*, grab.*, hand.move — image-space, 2D, boring on purpose.

text

hand-gesture (publisher)  →  ws://localhost  →  any consumer (canvas, agent, UI, …)

Sweep-to-clear, trash, fling, and the owner–helper lock stay in the canvas (or in whatever object model the consumer owns). Recognition never learns what a rectangle is.

That separation is the point. Here is another consumer — a desktop UI — listening to the same hand-gesture stream and driving its interface with hands. No vision code inside the app; only events.

A separate app consuming hand-gesture over WebSocket to drive its UI with hands

What I applied and why

Across the whole project — recognition, canvas, and the event boundary — the choices were the same kind of move: pick the smallest idea that survived a real failure.

$r = g/s$ instead of absolute gaps — so camera distance stops deciding whether a pinch counts.
$\tilde{r}=\operatorname{median}(\cdot)$ then hysteresis — estimate first, then decide enter/leave; one knob cannot do both jobs.
$\tau_{\mathrm{release}}(r_{\mathrm{held}})$ — a fixed OFF band punished loose pinches; the exit has to respect the gap you were holding.
Interaction redesign when projection collapses — pinch aimed at the camera can look like a fist; thresholds cannot invent separability, so empty-space vs on-figure rules do.
Owner / helper lock in the canvas — concurrent hands are a shared-resource problem at the object, not in the gesture stream.
Asymmetric fist release + $\mathbf{v}(t)=\mathbf{v}_0 e^{-t/\tau}$ — starting a grab stays deliberate; ending one and throwing should feel physical without living in the publisher.
hand-gesture as a publisher, not a dependency — a separate project that only emits events over WebSocket, so consumers never import vision and the two projects never couple process-to-process beyond the protocol.

The objective stays the same: hands manipulate objects abstractly; applications listen and decide what those objects are.

Open source: tomihq/hand-geometry-canvas.