Virtual Mouse Using Hand Gesture Recognition

Abstract

The Virtual Mouse Using Hand Gesture Recognition project presents a novel, marker-less human–computer interaction (HCI) system that converts everyday hand movements into standard cursor and mouse operations. Leveraging a single webcam, the system captures live video frames, preprocesses them with OpenCV, and extracts 21 hand landmarks via Google's MediaPipe Hands model. Custom Python routines analyze landmark configurations to distinguish between pointing, pinching, scrolling, left-clicking, right-clicking, freezing, and mode-toggle gestures. Detected gestures are translated into mouse events using PyAutoGUI, with mechanisms for smoothing cursor trajectories, debouncing rapid clicks, and intelligent mode toggling. The system features a multi-user environment where each user maintains independent, customizable gesture profiles stored in JSON configuration files, enabling personalized hand gesture mappings. A Flask-based web interface provides real-time gesture customization through camera capture and landmark storage, allowing users to record custom hand gestures for specific mouse actions and replace default hardcoded gestures. The implementation includes an adaptive default gesture system where the OK gesture intelligently toggles mouse mode only when both ON and OFF states use default gestures, while custom gestures take priority when either state is customized. Additionally, the system incorporates a sign language translator module capable of recognizing 9 distinct hand gestures (thumbs up/down, OK, stop, wave, peace, pistol, promise, point, and fist) in real-time, enabling sentence building for assistive communication. Extensive experimentation under varied indoor lighting demonstrates ≥85% gesture recognition accuracy, <100 ms end-to-end latency from hand detection to mouse action on mid-range hardware, and robust performance across three independent user profiles. The modular architecture employs process isolation through subprocess management, allowing concurrent execution of mouse control and sign language translation without interference. The implementation facilitates touch-free computing in medical, public kiosk, gaming, accessibility, and assistive communication contexts, removing dependence on physical peripherals, minimizing surface contact in contamination-sensitive environments, and providing inclusive HCI solutions for users with mobility constraints.