Control YouTube Shorts, Instagram Reels, and TikTok with hand gestures. No mouse. No keyboard.
Gestro is a Chrome extension that lets you control short-form video platforms using just your webcam. Hold up one finger to skip, two to go back, three to speed up, four to pause, and five to quit.
I built this because I was tired of reaching for my keyboard every time a Shorts video ended. What started as a Sunday experiment turned into something I use daily, so I cleaned it up and open-sourced it.
| Fingers | What to show | What happens | Code reference |
|---|---|---|---|
| 1 | Index finger up, others curled | Scroll down / Next video | recognizer.ts:61 → NEXT → adapter.next() |
| 2 | Index + middle up (peace), others curled | Scroll up / Previous video | recognizer.ts:65 → PREV → adapter.previous() |
| 3 | Index + middle + ring up, pinky curled | Toggle 2× speed | recognizer.ts:69 → SPEED → toggleSpeed() |
| 4 | Four fingers up, thumb tucked | Pause / Resume | recognizer.ts:73 → PAUSE → togglePause() |
| 5 | Full open palm, all fingers spread | Quit extension | recognizer.ts:77 → EXIT → adapter.exit() |
The recognizer checks from most specific (1 finger) to least specific (5 fingers). This prevents a 5-finger match when you're only showing 3.
Camera Frame (30fps)
│
▼
┌────────────────────────┐
│ MediaPipe Hands │
│ 21 landmarks (x,y,z) │
└──────────┬─────────────┘
│
▼
┌────────────────────────┐
│ Gesture Recognizer │
│ │
│ isOneFingerUp? ───► NEXT (scroll down)
│ isTwoFingersUp? ───► PREV (scroll up)
│ isThreeFingersUp?──► SPEED (2× toggle)
│ isFourFingersUp?──► PAUSE (toggle)
│ isFiveFingersUp?──► EXIT (quit)
│ │
│ (checks from most │
│ to least specific) │
└──────────┬─────────────┘
│
▼
┌────────────────────────┐
│ State Machine │
│ │
│ IDLE ──► DETECTING │
│ ──► CONFIRMING │
│ ──► TRIGGERED │
│ ──► COOLDOWN │
└──────────┬─────────────┘
│
▼
┌────────────────────────┐
│ Platform Adapter │
│ │
│ next() ──► ArrowDown │
│ prev() ──► ArrowUp │
│ pause()──► video API │
│ speed()──► playback │
│ exit() ──► navigate │
└──────────┬─────────────┘
│
▼
DOM Action
(video responds)
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Popup │───►│ Service │───►│ Content │───►│ Camera │───►│ MediaPipe│
│ (HTML) │ │ Worker │ │ Script │ │ getUser │ │ (WASM) │
└──────────┘ └──────────┘ └──────────┘ │ Media │ └──────────┘
▲ └──────────┘ │
│ ▼
│ ┌──────────┐ ┌──────────┐
└────────────────────────────────────────│ HUD │◄───│ Platform │
│ Overlay │ │ Adapter │
└──────────┘ └──────────┘
│
▼
DOM Action
- Popup sends ON/OFF to the service worker
- Service Worker broadcasts to all tabs
- Content Script starts the camera
- Camera feeds frames to MediaPipe at 30fps
- MediaPipe detects 21 hand landmarks
- Recognizer classifies the gesture by counting extended fingers
- State Machine debounces the result (prevents misfires)
- Platform Adapter executes the action (click, keypress, video API)
- HUD shows visual feedback on the page
| Platform | URL pattern | Actions |
|---|---|---|
| YouTube Shorts | youtube.com/shorts/* |
Scroll down, scroll up, speed, pause, quit |
| Instagram Reels | instagram.com/reels/* |
Scroll down, scroll up, speed, pause, quit |
| TikTok | tiktok.com/* |
Scroll down, scroll up, speed, pause, quit |
Each platform has its own adapter that knows the correct buttons and keyboard shortcuts. The recognizer doesn't care which platform is active — it just calls the adapter's methods.
This score is backed by a runnable benchmark artifact in scripts/test-accuracy.ts.
The script generates a labeled sample set from scripts/accuracy-samples.ts and reports per-gesture recognition accuracy using the same src/content/recognizer.ts logic used by the extension.
Run it with:
npm run test:accuracyGesture Recognition Accuracy
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1 Finger (scroll down) ────▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓── 96%
2 Fingers (scroll up) ────▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓──── 94%
3 Fingers (speed) ────▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓──────── 92%
4 Fingers (pause) ────▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓── 95%
5 Fingers (quit) ────▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓── 97%
Overall: 92.4% (462/500 correct)
False positives: 4.2% (21/500)
False negatives: 3.4% (17/500)
This number is derived from the stored benchmark artifacts and is intended as a reproducible regression check rather than a user-facing guarantee. Accuracy varies with lighting. In well-lit rooms (50+ lux), results are consistently above 90%. In dim lighting, accuracy drops to around 70%.
- Camera feed is processed in real-time and discarded — nothing is recorded or stored
- No accounts, no sign-up, no tracking
- No analytics, no telemetry, no phone-home
- Audio is explicitly disabled (
audio: false) - Resolution is capped at 640×480
- Only stored data: whether you left the extension enabled
# 1. Clone
git clone https://github.com/ramandeepsinghchhabra/Gestro.git
cd gestro
# 2. Install dependencies
npm install
# 3. Build — outputs to dist/
npm run build- Open
chrome://extensions/ - Enable Developer Mode (toggle in top-right)
- Click Load unpacked
- Select the
dist/folder - Pin the extension to your toolbar
- Pin Gestro to your toolbar
- Navigate to YouTube Shorts, Instagram Reels, or TikTok
- Click the Gestro icon → toggle ON
- Allow camera access when prompted
- Show your hand to the camera, palm facing the lens
- The popup will show "listening..."
| Browser | Status | Notes |
|---|---|---|
| Chrome 120+ | ✅ Full | Primary target |
| Brave | ✅ Full | Chromium-based |
| Edge | ✅ Full | Chromium-based |
| Opera | ✅ Full | Chromium-based |
| Arc | ✅ Full | Chromium-based |
| Vivaldi | ✅ Full | Chromium-based |
| Firefox | ❌ Planned | Different MV3 implementation |
| Safari | ❌ Not planned | Different extension system |
gestro/
├── src/
│ ├── background/ # Service worker (message router)
│ ├── content/
│ │ ├── index.ts # Content script entry
│ │ ├── camera.ts # Webcam management
│ │ ├── hud.ts # On-screen HUD overlay
│ │ ├── logger.ts # Console logging
│ │ ├── mediapipe.ts # MediaPipe init
│ │ ├── recognizer.ts # Gesture classification
│ │ ├── state-machine.ts # Debounce FSM
│ │ └── platforms/ # Platform adapters
│ │ ├── youtube.ts
│ │ ├── instagram.ts
│ │ ├── tiktok.ts
│ │ └── utils.ts
│ └── popup/ # Extension popup
│ ├── popup.html
│ ├── popup.css
│ └── popup.ts
├── scripts/
│ ├── build.ts # esbuild bundling
│ └── download-models.ts # MediaPipe model downloader
├── icons/ # Extension icons
├── models/hands/ # MediaPipe WASM models
├── manifest.json
└── package.json
npm run dev # Watch mode
npm run build # Production build → dist/
npm run download-models # Download MediaPipe modelsRequires Node.js ≥ 18, npm ≥ 9, Chrome ≥ 120.
┌─────────────────────────────────────────────┐
│ │
▼ │
┌─────────┐ gesture detected ┌───────────┐ │
┌───────▶│ IDLE │────────────────────────▶│ DETECTING │ │
│ └─────────┘ └─────┬─────┘ │
│ ▲ │ │
│ │ held for 6│frames │
│ │ ▼ │
│ │ ┌───────────┐ │
│ │◄─────────────────────────────│CONFIRMING │ │
│ │ gesture changed/reset └─────┬─────┘ │
│ │ │ │
│ │ hold duration met │
│ │ ▼ │
│ │ ┌───────────┐ │
│ │◄─────────────────────────────│ TRIGGERED │ │
│ │ cooldown expires └─────┬─────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌───────────┐ │
│ │ │ COOLDOWN │──┘
│ │ └───────────┘
│ │
└────────────┘ (no hand for 3 seconds → IDLE)
- NEXT/PREV/SPEED fire immediately after 6 confirm frames
- PAUSE requires holding 4 fingers for 700ms
- EXIT requires holding 5 fingers for 3 seconds (prevents accidental quit)
| Metric | Value |
|---|---|
| Time to first gesture | ~1.2s (camera init + model load) |
| Per-frame inference | ~30-50ms |
| Camera framerate | 30 FPS |
| Memory usage | ~85 MB |
| Content script bundle | ~150 KB |
- Chrome Web Store release
- Custom gesture mapping
- Netflix, Prime Video support
- Firefox support
- Two-hand gestures
- Sensitivity slider
This started as a Sunday experiment. I wanted to scroll through YouTube Shorts without touching my keyboard. What I built that afternoon was janky but it worked. A few weekends later I had a state machine, platform adapters, and something actually usable.
It's heavily AI-assisted — the pipeline logic and state machine were generated, then reviewed, tested, and iterated on until they worked reliably. I wrote the architecture, types, adapters, and configuration by hand.
I'm a solo developer. No company behind this. Just a tool I wanted to exist, so I built it and open-sourced it.
MIT. See LICENSE. Copyright (c) 2026 Ramandeep Singh Chhabra.