Efficient Training of Multi-Layer Neural Networks to Achieve Faster Validation

Artificial neural networks(ANNs)are one of the hottest topics in computer science and artificial intelligence due to their potential and advantages in analyzing real-world problems in various disciplines,including but not limited to physics,biology,chemistry,and engineering.However,ANNs lack several key characteristics of biological neural networks,such as sparsity,scale-freeness,and small-worldness.The concept of sparse and scale-free neural networks has been introduced to fill this gap.Network sparsity is implemented by removing weak weights between neurons during the learning process and replacing them with random weights.When the network is initialized,the neural network is fully connected,which means the number of weights is four times the number of neurons.In this study,considering that a biological neural network has some degree of initial sparsity,we design an ANN with a prescribed level of initial sparsity.The neural network is tested on handwritten digits,Arabic characters,CIFAR-10,and Reuters newswire topics.Simulations show that it is possible to reduce the number of weights by up to 50%without losing prediction accuracy.Moreover,in both cases,the testing time is dramatically reduced compared with fully connected ANNs.

A sparse algorithm for adaptive pruning least square support vector regression machine based on global representative point ranking

Least square support vector regression(LSSVR)is a method for function approximation,whose solutions are typically non-sparse,which limits its application especially in some occasions of fast prediction.In this paper,a sparse algorithm for adaptive pruning LSSVR algorithm based on global representative point ranking(GRPR-AP-LSSVR)is proposed.At first,the global representative point ranking(GRPR)algorithm is given,and relevant data analysis experiment is implemented which depicts the importance ranking of data points.Furthermore,the pruning strategy of removing two samples in the decremental learning procedure is designed to accelerate the training speed and ensure the sparsity.The removed data points are utilized to test the temporary learning model which ensures the regression accuracy.Finally,the proposed algorithm is verified on artificial datasets and UCI regression datasets,and experimental results indicate that,compared with several benchmark algorithms,the GRPR-AP-LSSVR algorithm has excellent sparsity and prediction speed without impairing the generalization performance.