自动化机器学习中的超参调优方法

Hyperparameter tuning methods in automated machine learning

本文介绍近年逐渐兴起的自动化机器学习框架,着重讨论其中颇具挑战的超参调优问题.常用的调参方法有格子点法、随机搜索等批量抽样策略,还包括Bayes优化、群体搜索算法和强化学习等序贯策略.由方开泰和王元早在1990年提出的贯序数论优化算法,利用序贯均匀设计对复杂响应曲面寻求全局最优值,同样适用于超参调优.本文以支持向量机和极限梯度推进机这两种常用的机器学习模型为例,结合两组典型的二分类数据集,对多种超参调优方法进行测试.通过比较分析发现一种改进的贯序数论优化算法,对解决自动化机器学习中的调参问题,颇具潜力.

In this paper,we make a comprehensive review of the challenging task of hyperparameter optimization in automated machine learning.The commonly used hyperparameter optimization methods include single-shot sampling strategies,e.g.,grid search,random search and sequential strategies where new trials are gradually augmented based on existing information,including Bayesian optimization,evolutionary algorithms,reinforcement learning-based methods.We find the sequential number-theoretic optimization(SNTO)algorithm proposed by Kai-Tai Fang and Yuan Wang in 1990 can also be applied in hyperparameter optimization,where sequential uniform designs are utilized to search the global optima in complex response surfaces.For illustration,various hyperparameter optimization methods are tested with two widely-used machine learning models including the support vector machine(SVM)and extreme gradient boosting(XGBoost),on two classical binary classification datasets.By analyzing the experimental results,we find a modified SNTO algorithm is quite promising in the hyperparameter optimization task.