PDF(Pubmed)
Abstract：
Deaf and hard-of-hearing people mainly communicate using sign language, which is a set of signs made using hand gestures combined with facial expressions to make meaningful and complete sentences. The problem that faces deaf and hard-of-hearing people is the lack of automatic tools that translate sign languages into written or spoken text, which has led to a communication gap between them and their communities. Most state-of-the-art vision-based sign language recognition approaches focus on translating non-Arabic sign languages, with few targeting the Arabic Sign Language (ArSL) and even fewer targeting the Saudi Sign Language (SSL). This paper proposes a mobile application that helps deaf and hard-of-hearing people in Saudi Arabia to communicate efficiently with their communities. The prototype is an Android-based mobile application that applies deep learning techniques to translate isolated SSL to text and audio and includes unique features that are not available in other related applications targeting ArSL. The proposed approach, when evaluated on a comprehensive dataset, has demonstrated its effectiveness by outperforming several state-of-the-art approaches and producing results that are comparable to these approaches. Moreover, testing the prototype on several deaf and hard-of-hearing users, in addition to hearing users, proved its usefulness. In the future, we aim to improve the accuracy of the model and enrich the application with more features.
摘要：
聋人和听力困难的人主要使用手语进行交流，这是一组符号，使用手势与面部表情相结合来制作有意义和完整的句子。聋人和听力障碍人士面临的问题是缺乏将手语翻译成书面或口头文本的自动工具，这导致了他们和社区之间的沟通差距。最先进的基于视觉的手语识别方法侧重于翻译非阿拉伯手语，很少有针对阿拉伯手语（ArSL）的，甚至更少的针对沙特手语（SSL）的。本文提出了一种移动应用程序，可以帮助沙特阿拉伯的聋人和听力障碍人士与他们的社区进行有效的沟通。该原型是一个基于Android的移动应用程序，应用深度学习技术将隔离的SSL转换为文本和音频，并包含其他针对ArSL的相关应用程序所没有的独特功能。拟议的方法，当在一个全面的数据集上评估时，通过超越几种最先进的方法并产生与这些方法相当的结果，证明了其有效性。此外，在几个聋哑和听力障碍用户身上测试原型，除了听力用户，证明了它的有用性。在未来,我们的目标是提高模型的准确性，并以更多的功能丰富应用。
