基于稳健设计的SGNS算法的超参数调优方法

Tuning method of hyper-parameters for SGNS algorithm based on robust design

通常给定超参数的若干取值选取性能最大的为最优组合(称为直接选优法),但是此方法的稳健性差。为此,提出了一种基于稳健设计的超参数调优方法(称为稳健调优法)。具体地,以SGNS算法中的超参数调优为例,在词语推断任务上实验并得出:经方差分析得到SGNS算法中的七个超参数中的五个对算法预测性能有显著影响,确定为主控因子,其余两个确定为噪声因子,且主控因子中有三个对性能估计的方差有显著影响,因此,调优中仅从期望最大来直接选优是不合理的;稳健调优法与直接选优法两者在预测性能上没有显著差异,但稳健调优法对噪声因子具有较好的稳健性。稳健调优法对一般的深度神经网络的调参有实际的借鉴意义。

Usually,the tuning method used commonly is to select the optimal combination of the values with the largest performance measure(called direct tuning method).However,this method has poor robustness.Hence,this paper proposed a robust hyper-parameter tuning method.Specifically,taking the tuning of SGNS(skip-gram with negative-sample)algorithm as an example in word analogy task,it drew the conclusions as follow.Five of all seven hyper-parameters in SGNS that had significant influence on the performance were determined as the control factors and remained two as the noise factors,and three of the five control factors had significant influence on the variance of the performance measure after ANOVA for experimental data.Therefore,direct selection the optimal combination only by maximum expectation was not reasonable.There was no significant difference in the prediction accuracy between robust tuning method and direct tuning method,but robust tuning method remarkably had good robustness.The robust tuning method had practical referential value for tuning hyper-parameters of deep neural networks.