Real-Time American Sign Language Recognition Using MediaPipe Hand Landmarks and Ensemble Machine Learning

V. Sai Subramanian, S. S. Karthik, J. Keerthna, Rejwan Bin Sulaiman, Prasanna Ranjith Christodoss, Ijaz Ahad Department of Artificial Intelligence and Machine Learning, SRM Institute of Science and Technology, Ramapuram, Chennai, Tamil Nadu, India. Department of Computer Science and Technology, Northumbria University, London, United Kingdom. Department of Computing, Mathematics and Physics, Messiah University, One University Ave, Mechanicsburg, Pennsylvania, United States of America. Department of Computer Systems Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China.

Abstract: This research paper introduces an extremely fast and efficient American Sign Language (ASL) understanding system that can work even if you don't have very high-end graphics processing hardware. Using MediaPipe Hands and Pose, developed by Google, this approach can extract hand positions and upper-body skeletal structure from the live camera feed. By combining these two, the system can create a very detailed 138-dimensional feature vector for each frame that captures not only hand movements but also body posture during signing. A Random Forest classifier trained on user-specific data is then used to determine the 26 ASL letters (A–Z) and 35 commonly used words and phrases. The system further uses a temporal smoothing buffer, which reduces short-term misclassifications but does not compromise responsiveness to rapid gesture changes. It can recognise 61 sign classes with 97.3% accuracy and execute real-time inference at 28–33 frames per second on a consumer device. Another option is an interactive OpenCV interface for sentence formation. This accessibility has a big influence. Experimental results reveal that landmark-based algorithms outperform prior pixel-intensity methods and enable robust generalisation over varied lighting circumstances, skin colours, and background scenes. A community-driven endeavour, the system enables deaf and hard-of-hearing people to connect with the hearing world efficiently and effectively. 

---

Keywords: American Sign Language (ASL); MediaPipe and Random Forest; Hand Landmarks; Real-Time Detection; Computer Vision; Gesture Recognition; Sign Language Recognition.

---

Received on: 16/07/2025Revised on: 29/09/2025Accepted on: 06/11/2025Published on: 03/03/2026

---

---