手语识别技术可以帮助有听力障碍的人与非听力障碍的人进行交流。目前,随着社会的快速发展,深度学习也为手语识别工作提供了一定的技术支持。在手语识别任务中,传统的卷积神经网络用于从手语视频中提取时空特征,导致识别率低。然而,大量基于视频的手语数据集需要大量的计算资源进行训练,同时确保网络的泛化,这对认可提出了挑战。在本文中,我们提出了一种基于残差网络(ResNet)和长短期记忆(LSTM)的基于视频的手语识别方法。随着网络层数量的增加,ResNet网络可以有效解决粒度爆炸问题,获得更好的时间序列特征。我们使用ResNet卷积网络作为骨干模型。LSTM利用门的概念来控制单元状态并更新序列的输出特征值。ResNet提取手语特征。然后,将学习到的特征空间作为LSTM网络的输入,获得长序列特征。它可以有效地提取手语视频中的时空特征,提高手语动作的识别率。广泛的实验评估证明了所提出方法的有效性和优越性能。准确率为85.26%,F1-得分为84.98%,阿根廷手语(LSA64)的准确率为87.77%。
Sign language recognition technology can help people with hearing impairments to communicate with non-hearing-impaired people. At present, with the rapid development of society, deep learning also provides certain technical support for sign language recognition work. In sign language recognition tasks, traditional convolutional neural networks used to extract spatio-temporal features from sign language videos suffer from insufficient feature extraction, resulting in low recognition rates. Nevertheless, a large number of video-based sign language datasets require a significant amount of computing resources for training while ensuring the generalization of the network, which poses a challenge for recognition. In this paper, we present a video-based sign language recognition method based on Residual Network (
ResNet) and Long Short-Term Memory (LSTM). As the number of network layers increases, the
ResNet network can effectively solve the granularity explosion problem and obtain better time series features. We use the
ResNet convolutional network as the backbone model. LSTM utilizes the concept of gates to control unit states and update the output feature values of sequences.
ResNet extracts the sign language features. Then, the learned feature space is used as the input of the LSTM network to obtain long sequence features. It can effectively extract the spatio-temporal features in sign language videos and improve the recognition rate of sign language actions. An extensive experimental evaluation demonstrates the effectiveness and superior performance of the proposed method, with an accuracy of 85.26%, F1-score of 84.98%, and precision of 87.77% on Argentine Sign Language (LSA64).