基于m×2正则化交叉验证的神经网络超参数调优方法

A Method for Hyper-parameter Tuning of Neural Network Based on m×2 Regularized Cross-validation

超参数调优是神经网络建模的关键问题。针对传统的超参数调优方法存在的问题,该文提出了一种基于m×2正则化交叉验证的超参数调优方法。目的是给出一种适用于复杂模型、大数据集背景下的计算开销较小且稳健的超参数调优方法。该方法的思想是从完整的数据集上选取少部分数据进行调优,避免模型在数据集较大时非常耗时的超参数调优难题;在m×2交叉验证的基础上设置正则化条件均衡训练集与验证集之间的分布差异,从而减少分布不一致带来的性能波动;使用信噪比作为调优的优化目标,从而可以综合考虑模型性能评价指标的均值和方差;并采用正交设计选择相关性较低的超参数组合以提高调优效率。以命名实体任务为例进行实验,在CoNLL 2003数据集上的实验结果显示,提出的调优方法能够选到和网格搜索性能上没有显著差异的超参数组合,且调优时间可显著降低约66%。

Hyper-parameter tuning is a key issue in neural network modeling.From the viewpoint of the problems of traditional hyper-parameter tuning methods,we propose a hyper-parameter tuning method based on m×2 regularized cross-validation.The goal is to present a robust hyper-parameter tuning method with low computational cost suitable for complex models and large datasets.The idea of the proposed method is to select a small number of data from the complete dataset for tuning,so as to avoid the time-consuming problem of hyper-parameter tuning when the dataset is large.Then,on the basis of m×2 cross-validation,regularization is adopted to balance the distribution difference between the training set and the validation set to reduce the performance fluctuation caused by the distribution inconsistency.The signal-to-noise ratio is used as the metric of hyper-parameter tuning,so that the mean and variance of the model performance can be comprehensively considered.The orthogonal design is used to select a combination of hyper-parameters with low correlation to improve the tuning efficiency.The experimental results on the CoNLL 2003 dataset show that the proposed method can obtain a combination of hyper-parameters that is not significantly different from the grid search,and the tuning time can be significantly reduced by about 66%.