学习长短时记忆网络的紧密结构

Learning the Compact Architecture of Long Short Term Memory Networks

近些年来,在语音识别任务上,前馈神经网络与长短时记忆网络等模型取得了突出的性能表现.然而,这些神经网络对其所要部署设备的内存资源与计算效率有较高的要求,这极大的阻碍了神经网络在移动设备上的应用.事实上大部分的神经网络都存在一定程度上的参数冗余,并由此带来了额外的计算量,因此压缩神经网络模型具有可行性与必要性.在这篇论文中,我们提出一种在网络训练过程中利用移动门来自动学习长短时记忆网络结构的方法,可以得到更加紧密的网络结构.在Sw itchboard上的实验结果显示我们提出的方法可以将长短时记忆网络的参数量减少到原来的58.7%,同时没有带来性能损失.

In the last few years,Feed Forward Neural Networks and Long Short Term Memory Networks have achieved state-of-art performance on many speech recognition tasks.However,these neural networks have higher demands for memory resources and computational efficiency of the devices that they are deploying,which hinders the application of neural networks on mobile devices.In fact,most of the neural networks have a certain degree of parameter redundancy and bring additional computation,and therefore,it is feasible and necessary to compress neural network models.In this paper,we propose a method to automatically learn the architectures of Long Short Term Memory Networks with moving gate during training,which achieves more compact architectures.Experimental results on the Switchboard task have shown that our proposed method can reduce the number of parameters in Long Short Term Memory Networks to 58.7% without performance loss.