02 — How it reads a hand

Twenty-one points, thirty times a second.

MediaPipe returns 21 landmarks per hand, each with x, y and a relative depth. Every gesture on this site is a handful of distances between them.

The landmark layout.

Point 0 is the wrist. Each finger runs outward in four points: base knuckle, two joints, then the tip. Index is 5 through 8, middle 9 through 12, ring 13 through 16, pinky 17 through 20, and the thumb is 1 through 4. Coordinates come back normalized to the frame, so 0.5 is the middle regardless of resolution.

The preview is mirrored so it behaves like a mirror rather than a photograph, which means x has to be flipped before it is used for anything directional. Otherwise moving right sends you backwards, and it is surprisingly hard to notice why.

Is a finger extended?

Compare the tip against the joint two below it. If the tip sits higher in the frame than that joint, the finger is up. Repeat for four fingers and count. The thumb needs its own test, because it folds sideways rather than down: compare how far the tip and the joint below it sit from the outer edge of the palm.

function fingersUp(lm){
  let n = 0;
  [[8,6],[12,10],[16,14],[20,18]].forEach(([tip, pip]) => {
    if (lm[tip].y < lm[pip].y) n++;
  });
  if (Math.abs(lm[4].x - lm[17].x) > Math.abs(lm[3].x - lm[17].x)) n++;
  return n;
}

Four or more counts as an open palm. Requiring all five makes the gesture brittle, because the pinky is the first thing to curl when you move quickly.

Is it a pinch?

Measure the distance from thumb tip to index tip, then divide by the distance from wrist to middle-finger knuckle. That second number is a decent proxy for how large your hand appears, so the ratio stays roughly constant as you move toward or away from the camera. Below the threshold, the pinch is closed.

const palm  = dist(lm[0], lm[9]);
const pinch = dist(lm[4], lm[8]) / palm < PINCH_RATIO;

Smoothing, and the trap in it.

Raw landmarks jitter by a pixel or two even when your hand is still, which reads as a slow drift when multiplied by the scroll gain. An exponential moving average fixes it: keep a smoothed position, and each frame move it halfway toward the new reading.

The trap is what you compare against afterwards. Subtracting the smoothed value from the raw one gives you the smoothing error, not the movement. The delta you actually want is between this frame's smoothed value and the previous frame's smoothed value, so the previous value has to be captured before the update overwrites it.

const prev = sy;
sy = prev === null ? y : prev + (y - prev) * 0.5;
const dy = prev === null ? 0 : sy - prev;   // the real movement

Velocity and the cooldown.

Horizontal velocity is the smoothed x delta divided by elapsed seconds, which makes the threshold independent of frame rate. Above the threshold with an open palm, the page changes and a cooldown starts. Without that lockout a single flick fires two or three times, because your hand is still travelling fast in the frames right after the trigger.

One more detail: reset the smoothed position to null whenever the hand leaves the frame. Otherwise your hand reappears somewhere new and the first delta is enormous, which lurches the page.

Tuning →