Fast Learning in Spiking Neural Networks by Learning Rate Adaptation

For accelerating the supervised learning by the SpikeProp algorithm with the temporal coding paradigm in spiking neural networks (SNNs), three learning rate adaptation methods (heuristic rule, delta-delta rule, and delta-bar-delta rule), which are used to speed up training in artificial neural networks, are used to develop the training algorithms for feedforward SNN. The performance of these algorithms is investigated by four experiments: classical XOR (exclusive or) problem, Iris dataset, fault diagnosis in the Tennessee Eastman process, and Poisson trains of discrete spikes. The results demonstrate that all the three learning rate adaptation methods are able to speed up convergence of SNN compared with the original SpikeProp algorithm. Furthermore, if the adaptive learning rate is used in combination with the momentum term, the two modifications will balance each other in a beneficial way to accomplish rapid and steady convergence. In the three learning rate adaptation methods, delta-bar-delta rule performs the best. The delta-bar-delta method with momentum has the fastest convergence rate, the greatest stability of training process, and the maximum accuracy of network learning. The proposed algorithms in this paper are simple and efficient, and consequently valuable for practical applications of SNN.

Tuning the Learning Rate for Stochastic Variational Inference

Stochastic variational inference （SVI） can learn topic models with very big corpora. It optimizes the variational objective by using the stochastic natural gradient algorithm with a decreasing learning rate. This rate is crucial for SVI; however, it is often tuned by hand in real applications. To address this, we develop a novel algorithm, which tunes the learning rate of each iteration adaptively. The proposed algorithm uses the Kullback-Leibler （KL） divergence to measure the similarity between the variational distribution with noisy update and that with batch update, and then optimizes the learning rates by minimizing the KL divergence. We apply our algorithm to two representative topic models： latent Dirichlet allocation and hierarchical Dirichlet process. Experimental results indicate that our algorithm performs better and converges faster than commonly used learning rates.

Improving the accuracy of heart disease diagnosis with an augmented back propagation algorithm

A multilayer perceptron neural network system is established to support the diagnosis for five most common heart diseases (coronary heart disease, rheumatic valvular heart disease, hypertension, chronic cor pulmonale and congenital heart disease). Momentum term, adaptive learning rate, the forgetting mechanics, and conjugate gradients method are introduced to improve the basic BP algorithm aiming to speed up the convergence of the BP algorithm and enhance the accuracy for diagnosis. A heart disease database consisting of 352 samples is applied to the training and testing courses of the system. The performance of the system is assessed by cross-validation method. It is found that as the basic BP algorithm is improved step by step, the convergence speed and the classification accuracy of the network are enhanced, and the system has great application prospect in supporting heart diseases diagnosis.

DP-ASSGD: Differential Privacy Protection Based on Stochastic Gradient Descent Optimization

Recently,differential privacy algorithms based on deep learning have become increasingly mature.Previous studies provide privacy mostly by adding differential privacy noise to the gradient,but it will reduce the accuracy,and it is difficult to balance privacy and accuracy.In this paper,the DP-ASSGD algo-rithm is proposed to counterpoise privacy and accuracy.The convergence speed is improved,the number of optimized iterations is decreased,and the privacy loss is significantly reduced.On the other hand,by using the postprocessing immunity characteristics of the differential privacy model,the Laplace smoothing mecha-nism is added to make the training process more stable and the generalization ability stronger.The experiment uses the MNIST dataset,with the same privacy budget,and compared with the existing differential privacy algorithms,the accu-racy is improved by 1.8%on average.When achieving the same accuracy,the DP-ASSGD algorithm consumes less privacy budget.

Adam revisited:a weighted past gradients perspective

Adaptive learning rate methods have been successfully applied in many fields,especially in training deep neural networks.Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients(i.e.,ADAM,RMSPROP)may fail to converge to the optimal solution.Though many algorithms,such as AMSGRAD and ADAMNC,have been proposed to fix the non-convergence issues,achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods.In this paper,we propose a novel adaptive method weighted adaptive algorithm(WADA)to tackle the non-convergence issues.Unlike AMSGRAD and ADAMNC,we consider using a milder growing weighting strategy on squared past gradient,in which weights grow linearly.Based on this idea,we propose weighted adaptive gradient method framework(WAGMF)and implement WADA algorithm on this framework.Moreover,we prove that WADA can achieve a weighted data-dependent regret bound,which could be better than the original regret bound of ADAGRAD when the gradients decrease rapidly.This bound may partially explain the good performance of ADAM in practice.Finally,extensive experiments demonstrate the effectiveness of WADA and its variants in comparison with several variants of ADAM on training convex problems and deep neural networks.

Speed up Training of the Recurrent Neural Network Based on Constrained optimization Techniques

In this paper, the constrained optimization technique for a substantial prob-lem is explored, that is accelerating training the globally recurrent neural net-work. Unlike most of the previous methods in feedforward neuxal networks, the authors adopt the constrained optimization technique to improve the gradiellt-based algorithm of the globally recuxrent neural network for the adaptive learn-ing rate during training. Using the recurrent network with the improved algo-rithm, some experiments in two real-world problems, namely filtering additive noises in acoustic data and classification of temporal signals for speaker identifi-cation, have been performed. The experimental results show that the recurrent neural network with the improved learning algorithm yields significantly faster training and achieves the satisfactory performance.

Real-time tool condition monitoring method based on in situ temperature measurement and artificial neural network in turning

Tool failures in machining processes often cause severe damages of workpieces and lead to large quantities of loss,making tool condition monitoring an important,urgent issue.However,problems such as practicability still remain in actual machining.Here,a real-time tool condition monitoring method integrated in an in situ fiber optic temperature measuring apparatus is proposed.A thermal simulation is conducted to investigate how the fluctuating cutting heats affect the measuring temperatures,and an intermittent cutting experiment is carried out,verifying that the apparatus can capture the rapid but slight temperature undulations.Fourier transform is carried out.The spectrum features are then selected and input into the artificial neural network for classification,and a caution is given if the tool is worn.A learning rate adaption algorithm is introduced,greatly reducing the dependence on initial parameters,making training convenient and flexible.The accuracy stays 90%and higher in variable argument processes.Furthermore,an application program with a graphical user interface is constructed to present real-time results,confirming the practicality.