基于多尺度特征融合的互学习脱机手写数学公式识别

Mutual Learning Offline Handwritten Mathematical Expression Recogni⁃tion Based on Multi-Scale Feature Fusion

脱机手写数学公式二维结构复杂,其中字符多变的尺度以及书写风格的变换不一都会增大手写数学公式识别的难度。文中提出了一个基于多尺度特征融合的互学习模型。首先,在编码阶段引入了多尺度特征融合的方式改进模型,以提升模型对公式中细粒度信息的提取能力以及加强对全局二维结构的语义信息理解;其次,引入了成对的手写体、打印体数据来进行互学习模型的训练,该模型包括解码器损失和上下文匹配损失,分别学习LaTeX语法以及手写体、打印体之间的语义不变性,提高模型对不同书写风格的鲁棒性,提升对公式整体信息的理解能力。在CROHME 2014/2016/2019数据集上进行实验验证,结果发现:引入多尺度特征融合机制后,表达式正确率分别达到55.25%、52.31%、53.72%;引入互学习机制后,表达式正确率分别达到55.43%、53.53%、53.79%;同时引入两种机制后,表达式正确率分别达到58.88%、55.10%、57.05%。经实验证明,文中提出的方法能够有效提取公式中不同尺度下的特征,并通过互学习机制克服手写风格不一、数据量少等问题。此外,在HME100K数据集上的实验结果也验证了文中提出模型的有效性。

With complex two-dimensional structure,offline handwritten mathematical expressions is difficult to recognize due to the variable scale of their symbols and the various transformation of their writing styles.This paper proposed a mutual learning model based on multi-scale feature fusion.Firstly,to enhance the model for extracting fine-grained information from expressions and comprehending semantic information of global two-dimensional structures,multi-scale feature fusion was introduced in the encoding stage.Secondly,paired handwritten and printed mathematical expressions were introduced for training the mutual learning model,which includes decoder loss and context matching loss to learn LaTeX grammar as well as semantic invariance between handwritten and printed mathematical expressions respectively to improve the robustness of the model to different writing styles.Experimental validation was performed on the CROHME 2014/2016/2019 dataset.After introducing the multiscale feature fusion mechanism,the expression correctness rate reaches 55.25%,52.31%,53.72%,respectively.After introducing the mutual learning mechanism,the expression correct rate reaches 55.43%,53.53%,53.79%,respectively.The expression correctness rate reaches 58.88%,55.10%,57.05% after introducing both mechanisms at the same time.It is proved experimentally that the proposed method can effectively extract the features in formulas at different scales and overcome the problems of different handwriting styles and small amount of data by mutual learning mechanism.In addition,the experimental results on the HME100K dataset verified the effectiveness of the proposed model.