Google DeepMind introduces SL2T sign-language-to-text translation model

Google DeepMind has introduced a massively multilingual sign-language-to-text translation model called SL2T. The model powers new sign language features in consumer products, starting with American Sign Language to English translation via sign-to-text dictation in Gboard and Live Transcribe on Pixel 11 devices, with support for more languages and devices coming soon.

Unlike traditional sign language technology that relied on gloves or intermediate text annotations called glosses, SL2T performs true machine translation. It uses an on-device model called MediaPipe Holistic to track body and hand pose landmarks for privacy, discarding the raw video feed immediately. The geometric coordinates are then sent to a server where SL2T translates the pose sequences directly into streaming text outputs.

The model is trained on over 100,000 hours of data across more than 50 sign languages. It achieves a zero-shot score of 70 BLEURT on the FLEURS-ASL benchmark, outperforming previous translation quality standards. Real-world applications were optimized to reduce streaming latency, prevent hallucinations on non-signing inputs, and account for left-handed and one-handed signers.

The project was developed in collaboration with the Deaf community, including guidance from a newly established AI Sign Language Advisory Committee. Google aims to expand the technology to additional sign languages and generation capabilities to improve digital accessibility for the estimated 70 million Deaf and hard of hearing people worldwide.