Gaze and Gesture Control

Hands-free laptop control from a webcam. Blink patterns, eye gaze, and hand gestures drive zooming, scrolling, clicking, and the mouse cursor - built for reading and navigating apps without touching the keyboard or trackpad.

Real-time computer vision with MediaPipe (face mesh + hands) and OpenCV; pyautogui sends the keyboard and mouse events. This is the eye/hand-control idea from the Apex notes, built as a standalone tool.

What it does

• Double blink -> zoom in, triple blink -> zoom out.
• Gaze cursor - toggle it with an open palm, then look around your resting point to slide the mouse. Look left of center and the cursor drifts left; hold the look to keep it moving. Press c to re-center the resting point after you shift in your seat.
• Pinch to drag - while the gaze cursor is on, a pinch holds the left mouse button so you can drag a scroll bar or slider by looking, then release the pinch to drop. Pinch is a distinct pose from an open palm (thumb and index touch, ring and pinky curled).
• Hand gestures -> actions, listed below and rebindable in gestures.json.
• The window overlays tracking dots (green eye contour, magenta iris, orange hand skeleton) and a side panel with a live gaze pad (rest cross, deadzone box, current-offset dot), the current settings, and a running gesture log. The window is resizable; press f for full screen.
• Pause with the space bar to freeze all control and use the mouse normally; press it again to resume.
• Press q or Esc to quit (settings are saved on exit).

Default gesture bindings

Input	Action
Double blink	Zoom in (Ctrl +)
Triple blink	Zoom out (Ctrl -)
Open palm	Toggle gaze cursor on/off
Pinch (thumb + index)	Hold left button to drag (release to drop)
Fist	Left click
Point (index only)	Right click
Peace (index + middle)	Scroll up
Three fingers	Scroll down
Thumbs up	Volume up

Custom bindings

gestures.json maps each blink pattern and gesture name to an action:

{
  "blink": { "double": "zoom_in", "triple": "zoom_out" },
  "hands": {
    "fist": "left_click",
    "open_palm": "toggle_gaze",
    "peace": "scroll_up",
    "three": "scroll_down",
    "thumbs_up": "volume_up",
    "point": "right_click"
  }
}

Pinch is handled directly as a drag and is not listed here. Rebind any of the others by editing the action on the right. Available actions live in src/actions.py: zoom_in, zoom_out, scroll_up, scroll_down, left_click, right_click, double_click, volume_up, volume_down, next_tab, prev_tab, new_tab, close_tab, switch_window, show_desktop, page_up, page_down, copy, paste, undo, redo, screenshot, plus the special toggle_gaze (pinch-to-drag is wired in main.py, not through this list). Edit gestures.json and press r in the app to reload bindings without restarting. Add your own action by writing a function in src/actions.py and adding it to the ACTIONS dict. Recognizing a brand-new hand shape means adding a rule to classify in src/gestures.py; the recognized names are fist, open_palm, peace, three, point, thumbs_up, and pinch.

Setup

MediaPipe 0.10.14 needs protobuf < 5, so use the pinned requirements.txt in a clean virtual environment:

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
python main.py

A window shows the mirrored camera with tracking dots and the status panel. Good, even lighting and the whole face in frame help detection.

Tuning

Sensitivity and timing are adjusted live from the keyboard and saved to settings.json on exit (or press s to save immediately). Defaults live in src/settings.py; delete settings.json to reset.

Key	Effect
Space	Pause / resume all control
g	Toggle the gaze cursor
c	Re-center the gaze rest point
f	Toggle full screen
+ / -	Gaze speed up / down
[ / ]	Deadzone smaller / larger
< / >	Gesture cooldown shorter / longer
r	Reload gestures.json bindings
s	Save settings now
q / Esc	Quit (saves settings)

Notes

• pyautogui's fail-safe is disabled so the gaze cursor can reach screen corners; quit with q/Esc or turn the gaze cursor off with an open palm.
• Gaze uses the iris position inside each eye box, so it is head-position independent, but it is still calibration-free and velocity-based - a coarse pointer, not pixel-accurate absolute tracking. The rest point does not drift, so a held look keeps moving; press c to re-center it if you shift position. Tune gain/deadzone/max_step in src/cursor.py to your camera; the vertical gain is higher because the eye is shorter than it is wide.
• Hand gestures confirm over STABLE_FRAMES frames and then wait COOLDOWN seconds (main.py) so one pose fires once - lower these if it feels sluggish, raise them if a gesture double-fires.
• Blink thresholds live in src/blink.py.

Layout

main.py            capture loop, event dispatch, on-screen HUD
gestures.json      blink/gesture -> action bindings
src/detectors.py   MediaPipe face-mesh + hands, per-frame landmarks
src/blink.py       eye-aspect-ratio blink clustering (double/triple)
src/gestures.py    hand-pose classification
src/cursor.py      gaze-to-cursor velocity control
src/actions.py     the pyautogui actions and their registry
src/config.py      loads gestures.json
src/settings.py    load/save the runtime-tunable settings
src/overlay.py     tracking dots, gaze pad, and the status/log panel