Cameras and AI as sign language interpreters

Here’s a quiet revolution you can hold in your hands: a laptop webcam, a bit of clever AI, and the possibility of an easier conversation between Deaf signers and hearing nonsigners. In a paper published in Sensors, a team led by Bader Alsharif, working with colleagues Easa Alalwany, Ali Ibrahim, Imad Mahgoub, and Mohammad Ilyas across Florida Atlantic University in Boca Raton, Taibah University in Saudi Arabia, and the Technical and Vocational Training Corporation in Riyadh, describe a real-time American Sign Language (or ASL) interpreter that runs on everyday hardware. It recognizes the alphabet as you fingerspell, then turns those gestures into text, fast enough to keep a conversation going.

Why this matters is bigger than a neat demo. In the U.S., hearing loss touches tens of millions of people, and a meaningful share identify as Deaf or report serious difficulty hearing, communities that regularly run into communication barriers at school, at work, and in medical and civic life. Authoritative health data estimate that about 1 in 8 people (roughly 30 million) ages 12 and older have hearing loss in both ears. The need for human professionals is real, but unfortunately, there’s a shortage of qualified ASL interpreters, especially outside major cities and on college campuses. Technology is not a replacement for interpreters. However, tools like this can fill gaps, support everyday interactions, and lower the barrier to basic access.

What the team built

Think of the system as two dance partners moving in sync. First, a tracker that recognizes where the hands are in every video frame based on the position of the wrist, knuckles, and fingertips, creating a skeleton of points. Second, a sign detector based on YOLOv11, one among the latest generation of a family of models for everyday computer vision.

The result is a system that watches your hands through a regular webcam, locks onto the structure of your fingers, and classifies each static ASL letter in real time. Fingerspelling can be tricky because many letters look deceptively similar. So, the authors trained on a large, varied collection of images and used the fingertip-level landmarks to guide their detector toward the details that matter (like how the thumb wraps for “A” vs. “T”).

Why fingerspelling first?

Language is rich and fluid; ASL is no exception. Continuous sign language (full words and sentences) blends motion, shape, facial expression, and context. That’s a high bar for any automated system. Starting with the alphabet, static handshapes used to spell names, places, and uncommon words, lets researchers deliver something useful right away: spelling a last name at a clinic desk, typing an address into a phone, clarifying a brand or technical term in a meeting. The prototype discussed in the paper focuses on this practical core and, importantly, runs without exotic equipment, as the system uses a built-in camera.

Under the hood

Understanding how the system works is relatively simple. Imagine you’re teaching a camera to be a good listener. The tracker finds the same 21 spots on the hand over and over, like following the joints of a marionette. Those points are normalized so the system doesn’t care if your hand is close to or far from the lens. Instead of retraining everything from scratch, the team uses transfer learning: they take a system already good at seeing edges and shapes and fine-tune it on the specifics of ASL handshapes. That keeps training efficient and makes the model less fragile to lighting or background changes. The researchers assembled a large, balanced set of hand images with different skin tones, orientations, and lighting, and they augmented it with small tweaks (like gentle rotations or brightness changes). The goal: a model that isn’t thrown off by the messy reality of coffee shops, classrooms, and dim living rooms.

What can it do today?

In the lab setting described by the authors, the system recognizes the ASL alphabet letters quickly and reliably, assembling them into text so users can spell names and locations. The everyday-hardware requirement is key because it hints at how such tools could show up as apps, browser extensions, or kiosk software, places where you don’t get to control the environment. The paper emphasizes responsiveness (low delay between gesture and text) and robustness (steady performance under different lighting and backgrounds), which are exactly the knobs you need to turn for real-world use.

No single system solves communication access, not for hearing people, and not for Deaf people. In fact, this prototype does not translate full ASL sentences. Continuous signing involves motion over time, grammar that isn’t English, and expression beyond the hands. Also, it won’t replace human interpreters. Live interpreters bring cultural and linguistic expertise, handle nuance, and manage complex settings that an app can’t replicate.

However, for now, this system can be a pocketable tool that lets two people share a name, an address, or a few clarifying words without waiting for help.

So, what would it take to move from fingerspelling to full signing? Time-aware programs that reason about sequences of signs would let a system track handshapes and the motion between them. Also, ASL grammar uses facial expressions, head movements, and body posture. Incorporating face and body cues will be crucial. Finally, the availability of community-sourced, consented video corpora representing many signers, dialects, and settings would help models generalize and reduce bias, which is an ongoing challenge in sign language technology research.

If you want to learn more, the original article titled "Real-Time American Sign Language Interpretation Using Deep Learning and Keypoint Tracking" on Sensors at https://doi.org/10.3390/s25072138.