Abstract：
Continuous Sign Language Recognition (CSLR) is a task which converts a sign language video into a gloss sequence. The existing deep learning based sign language recognition methods usually rely on large-scale training data and rich supervised information. However, current sign language datasets are limited, and they are only annotated at sentence-level rather than frame-level. Inadequate supervision of sign language data poses a serious challenge for sign language recognition, which may result in insufficient training of sign language recognition models. To address above problems, we propose a cross-modal knowledge distillation method for continuous sign language recognition, which contains two teacher models and one student model. One of the teacher models is the Sign2Text dialogue teacher model, which takes a sign language video and a dialogue sentence as input and outputs the sign language recognition result. The other teacher model is the Text2Gloss translation teacher model, which targets to translate a text sentence into a gloss sequence. Both teacher models can provide information-rich soft labels to assist the training of the student model, which is a general sign language recognition model. We conduct extensive experiments on multiple commonly used sign language datasets, i.e., PHOENIX 2014T, CSL-Daily and QSL, the results show that the proposed cross-modal knowledge distillation method can effectively improve the sign language recognition accuracy by transferring multi-modal information from teacher models to the student model. Code is available at https://github.com/glq-1992/cross-modal-knowledge-distillation_new.
摘要：
连续手语识别（CSLR）是将手语视频转换为光泽序列的任务。现有的基于深度学习的手语识别方法通常依赖于大规模的训练数据和丰富的监督信息。然而,当前手语数据集有限，它们仅在句子级别而不是框架级别进行注释。对手语数据的监管不足对手语识别提出了严峻的挑战,这可能导致手语识别模型训练不足。为了解决上述问题，我们提出了一种用于连续手语识别的跨模态知识蒸馏方法，其中包含两个教师模型和一个学生模型。教师模型之一是Sign2Text对话教师模型，输入手语视频和对话句，输出手语识别结果。另一种教师模式是Text2Gloss翻译教师模式，其目标是将文本句子翻译成光泽序列。两种教师模式都可以提供信息丰富的软标签来辅助学生模式的训练,这是一个通用的手语识别模型。我们对多个常用的手语数据集进行了广泛的实验，即，凤凰2014T,CSL-Daily和QSL,结果表明,所提出的跨模态知识蒸馏方法通过将多模态信息从教师模型传递到学生模型,能够有效提高手语识别的准确率。代码可在https://github.com/glq-1992/cross-modal-knowledge-restrination_new上找到。
