Tuning the Learning Rate for Stochastic Variational Inference

Stochastic variational inference （SVI） can learn topic models with very big corpora. It optimizes the variational objective by using the stochastic natural gradient algorithm with a decreasing learning rate. This rate is crucial for SVI; however, it is often tuned by hand in real applications. To address this, we develop a novel algorithm, which tunes the learning rate of each iteration adaptively. The proposed algorithm uses the Kullback-Leibler （KL） divergence to measure the similarity between the variational distribution with noisy update and that with batch update, and then optimizes the learning rates by minimizing the KL divergence. We apply our algorithm to two representative topic models： latent Dirichlet allocation and hierarchical Dirichlet process. Experimental results indicate that our algorithm performs better and converges faster than commonly used learning rates.

DP-ASSGD: Differential Privacy Protection Based on Stochastic Gradient Descent Optimization

Recently,differential privacy algorithms based on deep learning have become increasingly mature.Previous studies provide privacy mostly by adding differential privacy noise to the gradient,but it will reduce the accuracy,and it is difficult to balance privacy and accuracy.In this paper,the DP-ASSGD algo-rithm is proposed to counterpoise privacy and accuracy.The convergence speed is improved,the number of optimized iterations is decreased,and the privacy loss is significantly reduced.On the other hand,by using the postprocessing immunity characteristics of the differential privacy model,the Laplace smoothing mecha-nism is added to make the training process more stable and the generalization ability stronger.The experiment uses the MNIST dataset,with the same privacy budget,and compared with the existing differential privacy algorithms,the accu-racy is improved by 1.8%on average.When achieving the same accuracy,the DP-ASSGD algorithm consumes less privacy budget.

Adam revisited:a weighted past gradients perspective

Adaptive learning rate methods have been successfully applied in many fields,especially in training deep neural networks.Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients(i.e.,ADAM,RMSPROP)may fail to converge to the optimal solution.Though many algorithms,such as AMSGRAD and ADAMNC,have been proposed to fix the non-convergence issues,achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods.In this paper,we propose a novel adaptive method weighted adaptive algorithm(WADA)to tackle the non-convergence issues.Unlike AMSGRAD and ADAMNC,we consider using a milder growing weighting strategy on squared past gradient,in which weights grow linearly.Based on this idea,we propose weighted adaptive gradient method framework(WAGMF)and implement WADA algorithm on this framework.Moreover,we prove that WADA can achieve a weighted data-dependent regret bound,which could be better than the original regret bound of ADAGRAD when the gradients decrease rapidly.This bound may partially explain the good performance of ADAM in practice.Finally,extensive experiments demonstrate the effectiveness of WADA and its variants in comparison with several variants of ADAM on training convex problems and deep neural networks.

Speed up Training of the Recurrent Neural Network Based on Constrained optimization Techniques

In this paper, the constrained optimization technique for a substantial prob-lem is explored, that is accelerating training the globally recurrent neural net-work. Unlike most of the previous methods in feedforward neuxal networks, the authors adopt the constrained optimization technique to improve the gradiellt-based algorithm of the globally recuxrent neural network for the adaptive learn-ing rate during training. Using the recurrent network with the improved algo-rithm, some experiments in two real-world problems, namely filtering additive noises in acoustic data and classification of temporal signals for speaker identifi-cation, have been performed. The experimental results show that the recurrent neural network with the improved learning algorithm yields significantly faster training and achieves the satisfactory performance.

Real-time tool condition monitoring method based on in situ temperature measurement and artificial neural network in turning

Tool failures in machining processes often cause severe damages of workpieces and lead to large quantities of loss,making tool condition monitoring an important,urgent issue.However,problems such as practicability still remain in actual machining.Here,a real-time tool condition monitoring method integrated in an in situ fiber optic temperature measuring apparatus is proposed.A thermal simulation is conducted to investigate how the fluctuating cutting heats affect the measuring temperatures,and an intermittent cutting experiment is carried out,verifying that the apparatus can capture the rapid but slight temperature undulations.Fourier transform is carried out.The spectrum features are then selected and input into the artificial neural network for classification,and a caution is given if the tool is worn.A learning rate adaption algorithm is introduced,greatly reducing the dependence on initial parameters,making training convenient and flexible.The accuracy stays 90%and higher in variable argument processes.Furthermore,an application program with a graphical user interface is constructed to present real-time results,confirming the practicality.