手语识别与翻译综述

Survey of Sign Language Recognition and Translation

不同于有声语言,手语主要由连续的手势动作构成。手语识别与翻译是促成听障人士与健听人士之间无障碍交流的重要手段。手语识别与翻译研究任务通过对手语视频进行处理分析并以文字形式显示识别结果,是一种典型的多领域交叉研究。近年来,基于深度学习的手语识别与翻译研究获得了长足的进步。为了便于研究者们系统、全面地了解手语识别与翻译研究任务,分别以手语识别和手语翻译两大任务为主线,从三方面展开综述工作:首先,对具备代表性的手语识别和手语翻译研究工作进行分类总结并分析其特点;其次,归纳整理当前常用的不同国别手语识别与翻译研究数据集,分别从孤立词和连续手语语句两个角度进行分类,同时根据手语识别和手语翻译研究任务的差异性,介绍了对应的评价指标体系;最后,从手语视觉特征的有效信息提取、多线索权重分配、手语与自然语言语法对应及手语数据集资源等方面总结了手语识别与翻译研究目前存在的主要挑战。

Different from spoken languages,sign language is mainly composed of continuous gestures.Sign language recognition and translation are important means of facilitating barrier-free communication between the hearingimpaired and the hearing person.The sign language recognition and translation research task is a typical multidomain cross-study by processing and analyzing sign language videos and displaying the recognition results in text form.In recent years,sign language recognition and translation research based on deep learning has made great progress.In order to facilitate researchers to systematically and comprehensively understand the research tasks of sign language recognition and translation,the review work is carried out from the perspectives of sign language recognition and sign language translation.Firstly,the translation research work is classified and summarized and its characteristics are analyzed.Secondly,the common sign language recognition and translation research datasets of different countries are summarized and classified from the perspectives of isolated sign language words and continuous sign language sentences.Based on the difference in research tasks,the corresponding evaluation index system is introduced.Finally,the major challenges of current research on sign language recognition and translation are summarized from the aspects of effective information extraction of sign language visual features,multi-cue weight assignment,relationship between sign language and natural language grammar,and sign language dataset resources.