基于零阶减小方差方法的鲁棒支持向量机

Robust SVM Based on Zeroth Order Variance Reduction

采用传统的支持向量机方法对含有噪声的数据进行分类时会产生较大的损失,使得分类超平面严重偏离最优超平面,从而导致分类性能较差。为了解决此问题,文中提出了一种鲁棒的支持向量机(Robust Support Vector Machine,RSVM)方法,该方法给出了一种正弦平方形式的损失函数,根据正弦函数的特点,即使对于噪声数据,其损失函数的值也会被限制在[0,1]区间,从而提高了支持向量机的抗噪性。另外,在求解支持向量机时,传统的随机梯度下降方法在每次迭代中利用单个样本梯度近似代替全梯度,这样必然会产生方差,而随着迭代次数的增加,方差也不断累积,从而严重影响算法的分类性能。为了减小方差的影响,引入零阶减小方差的随机梯度下降(Zeroth Order-Stochastic Variance Reduced Gradient,ZO-SVRG)算法。该算法使用坐标梯度估计方法近似代替梯度,通过在每轮迭代中引入梯度修正项来减小方差的影响;同时,采取加权平均的输出形式进行内外循环的输出,加快了优化问题的收敛速度。实验结果表明,提出的基于零阶减小方差方法的鲁棒支持向量机算法对噪声数据具有更好的鲁棒性,且有效降低了方差的影响;为了进一步提高算法的性能,对实验中主要参数λ,k对算法精度的影响进行了分析。对于线性和非线性两种情况,当其参数对(λ,k)分别满足(λ=1,k=5)和(λ=10,k=3)时,可以达到各自的最高精度。

Great losses will be produced when traditional SVM methods are used to deal with the classification problem with noisy data,which makes the classification hyperplane seriously deviates from the optimal hyperplane,resulting in poor classification performance.In order to solve this problem,this paper proposed a robust support vector machine(RSVM)and gave a loss function in sinusoidal square form.According to the characteristics of sinusoidal function,the value of loss function is limited to the range of[0,1],even for noise data,which improves the anti-noise ability of SVM.In addition,when the traditional stochastic gradient descent method is used to solve the SVM,a single sample gradient is used to approximately replace the full gradient in each iteration,which will inevitably produce variance.As the number of iterations increases,the variance also accumulates,which seriously affects the classification performance of the algorithm.In order to reduce the influence of variance,this paper introduced a zeroth order-stochastic variance reduced gradient(ZO-SVRG)algorithm.This algorithm uses coordinate gradient estimation method to replace gradient approximately,and reduces the influence of variance by introducing the gradient correction term in each iteration.Besides,in the output of the internal and external loop,the weighted average output form is adopted,and then the convergence speed of the optimization problem is accelerated.The experimental results show that the robust support vector machine based on zeroth-order variance reduction algorithm has better robustness to noise data and effectively reduces the influence of variance.In order to further improve the performance of the algorithm,the influence of the main parametersλand k on the accuracy of algorithm were analyzed.For both linear and nonlinear cases,when its parameter pairs(λ,k)are satisfied(λ=1,k=5)and(λ=10,k=3),respectively,the highest accuracy of each can be achieved.