Adaptive learning rate methods have been successfully applied in many fields,especially in training deep neural networks.Recent results have shown that adaptive methods with exponential increasing weights on squared p...Adaptive learning rate methods have been successfully applied in many fields,especially in training deep neural networks.Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients(i.e.,ADAM,RMSPROP)may fail to converge to the optimal solution.Though many algorithms,such as AMSGRAD and ADAMNC,have been proposed to fix the non-convergence issues,achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods.In this paper,we propose a novel adaptive method weighted adaptive algorithm(WADA)to tackle the non-convergence issues.Unlike AMSGRAD and ADAMNC,we consider using a milder growing weighting strategy on squared past gradient,in which weights grow linearly.Based on this idea,we propose weighted adaptive gradient method framework(WAGMF)and implement WADA algorithm on this framework.Moreover,we prove that WADA can achieve a weighted data-dependent regret bound,which could be better than the original regret bound of ADAGRAD when the gradients decrease rapidly.This bound may partially explain the good performance of ADAM in practice.Finally,extensive experiments demonstrate the effectiveness of WADA and its variants in comparison with several variants of ADAM on training convex problems and deep neural networks.展开更多
The decision-making units(DMUs)in the modern service industries may produce desirable outputs and undesirable outputs.For the decision makers,some outputs may be more desired than others although all of them are desir...The decision-making units(DMUs)in the modern service industries may produce desirable outputs and undesirable outputs.For the decision makers,some outputs may be more desired than others although all of them are desirable.Considering these characteristics,this work combines the data envelopment analysis(DEA)and the multiple attributes decision-making(MADM)method,to make a reasonable and comprehensive performance evaluation for DMUs.Specifically,three DEA-based models are modified to obtain more reasonable efficiency scores for DMUs.The MADM method is used to determine the weights of outputs based on the preference ratings within the outputs.The efficiency scores are then multiplied by the aggregated outputs quantities to obtain the comprehensive performance scores for evaluation.The effectiveness of the proposed models is demonstrated by extensive numerical experiments.展开更多
基金We thank the anonymous reviewers for their insightful comments and discussions.This research was partially supported by grants from the National Key Research and Development Program of China(2018YFB1004300)the National Natural Science Foundation of China(Grant Nos.61703386,61727809,and U1605251).
文摘Adaptive learning rate methods have been successfully applied in many fields,especially in training deep neural networks.Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients(i.e.,ADAM,RMSPROP)may fail to converge to the optimal solution.Though many algorithms,such as AMSGRAD and ADAMNC,have been proposed to fix the non-convergence issues,achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods.In this paper,we propose a novel adaptive method weighted adaptive algorithm(WADA)to tackle the non-convergence issues.Unlike AMSGRAD and ADAMNC,we consider using a milder growing weighting strategy on squared past gradient,in which weights grow linearly.Based on this idea,we propose weighted adaptive gradient method framework(WAGMF)and implement WADA algorithm on this framework.Moreover,we prove that WADA can achieve a weighted data-dependent regret bound,which could be better than the original regret bound of ADAGRAD when the gradients decrease rapidly.This bound may partially explain the good performance of ADAM in practice.Finally,extensive experiments demonstrate the effectiveness of WADA and its variants in comparison with several variants of ADAM on training convex problems and deep neural networks.
基金This work was supported by Science and Technology Foundation of Jiangxi Educational Committee[grant number GJJ190287].
文摘The decision-making units(DMUs)in the modern service industries may produce desirable outputs and undesirable outputs.For the decision makers,some outputs may be more desired than others although all of them are desirable.Considering these characteristics,this work combines the data envelopment analysis(DEA)and the multiple attributes decision-making(MADM)method,to make a reasonable and comprehensive performance evaluation for DMUs.Specifically,three DEA-based models are modified to obtain more reasonable efficiency scores for DMUs.The MADM method is used to determine the weights of outputs based on the preference ratings within the outputs.The efficiency scores are then multiplied by the aggregated outputs quantities to obtain the comprehensive performance scores for evaluation.The effectiveness of the proposed models is demonstrated by extensive numerical experiments.
