基于神经网络语言模型的动态层序Softmax训练算法

Training algorithm of dynamic hierarchical Softmax based on neural network language model

针对词向量训练过程中层序Softmax算法无法进行增量训练及海量数据训练低效的问题,提出了动态层序Softmax算法.通过对数据样本的增量加载,采用结点置换方法动态构建编码树,实现对样本的增量训练.为避免损失函数因样本量较少而呈现震荡式下降,利用梯度的一阶矩估计与二阶矩估计动态调整参数更新方向与学习率,通过梯度迭代缩小权值变化范围和收敛训练误差,提高词向量的训练效率.以维基百科中文语料作为数据进行了试验,完成了训练效率和质量的分析.结果表明:相较于现有方法动态层序Softmax算法显著提高了训练效率,当增量样本大小为10 kB^1 MB时,训练增速有近30倍的提升,有效地缩短训练周期.

To solve the problems of hierarchical Softmax algorithm in the training process of word vectors with inability of incremental training and inefficient training of massive data,the dynamic hierarchical Softmax algorithm was proposed.By the incremental loading of data samples,an adaptive Huffman coding tree was dynamically constructed by the node adjustment replacement method.To avoid the oscillatory decline of loss function due to the small sample size,the first-order and the second-order moment estimations of the gradient were used to dynamically adjust the parameters update direction and learning rate.The weight variation range and the convergence training network error were reduced by the gradient descent algorithm to improve the training efficiency of the word vector from massive data.The Wikipedia Chinese corpus was adopted as the data to test the training efficiency and quality.The experimental results show that the dynamic hierarchical Softmax algorithm can significantly improve the training efficiency and ensure the quality of word vector training.When the incremental samples are from 10 kB to 1 MB,the training speed is increased about 30 times,which can effectively shorten the training period.