Increasing Momentum-Like Factors:A Method for Reducing Training Errors on Multiple GPUs

In distributed training,increasing batch size can improve parallelism,but it can also bring many difficulties to the training process and cause training errors.In this work,we investigate the occurrence of training errors in theory and train ResNet-50 on CIFAR-10 by using Stochastic Gradient Descent(SGD) and Adaptive moment estimation(Adam) while keeping the total batch size in the parameter server constant and lowering the batch size on each Graphics Processing Unit(GPU).A new method that considers momentum to eliminate training errors in distributed training is proposed.We define a Momentum-like Factor(MF) to represent the influence of former gradients on parameter updates in each iteration.Then,we modify the MF values and conduct experiments to explore how different MF values influence the training performance based on SGD,Adam,and Nesterov accelerated gradient.Experimental results reveal that increasing MFs is a reliable method for reducing training errors in distributed training.The analysis of convergent conditions in distributed training with consideration of a large batch size and multiple GPUs is presented in this paper.

Reduce Training Error of Extreme Learning Machine by Selecting Appropriate Hidden Layer Output Matrix

Extreme learning machine(ELM)is a feedforward neural network with a single layer of hidden nodes,where the weight and the bias connecting input to hidden nodes are randomly assigned.The output weight between hidden nodes and outputs are learned by a linear model.It is interesting to ask whether the training error of ELM is significantly affected by the hidden layer output matrix H,because a positive answer will enable us obtain smaller training error from better H.For single hidden layer feedforward neural network(SLFN)with one input neuron,there is significant difference between the training errors of different Hs.We find there is a reliable strong negative rank correlation between the training errors and some singular values of the Moore-Penrose generalized inverse of H.Based on the rank correlation,a selection algorithm is proposed to choose robust appropriate H to achieve smaller training error among numerous Hs.Extensive experiments are carried out to validate the selection algorithm,including tests on real data set.The results show that it achieves better performance in validity,speed and robustness.