Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

We prove,under mild conditions,the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model,both in batch gradient descent and stochastic gradient descent.We also discuss a Riemannian version of the Adam algorithm.We show numerical simulations of these algorithms on various benchmarks.

Neural network based on adaptive chaotic gradient descending optimization algorithm and its application in matte converting process

An adaptive chaotic gradient descending optimization algorithm for single objective optimization was presented. A local minimum judged by two rules was obtained by an improved mutative-step gradient descending method. A new optimal minimum was obtained to replace the local minimum by mutative-scale chaotic search algorithm whose scales are magnified gradually from a small scale in order to escape local minima. The global optimal value was attained by repeatedly iterating. At last, a BP (back-propagation) neural network model for forecasting slag output in matte converting was established. The algorithm was used to train the weights of the BP neural network model. The simulation results with a training data set of 400 samples show that the training process can be finished within 300 steps to obtain the global optimal value, and escape local minima effectively. An optimization system for operation parameters, which includes the forecasting model, is achieved, in which the output of converter increases by 6.0%, and the amount of the treated cool materials rises by 7.8% in the matte converting process.

Online Gradient Methods with a Punishing Term for Neural Networks

Online gradient methods are widely used for training the weight of neural networks and for other engineering computations. In certain cases, the resulting weight may become very large, causing difficulties in the implementation of the network by electronic circuits. In this paper we introduce a punishing term into the error function of the training procedure to prevent this situation. The corresponding convergence of the iterative training procedure and the boundedness of the weight sequence are proved. A supporting numerical example is also provided.

A Study on the Convergence of Gradient Method with Momentum for Sigma-Pi-Sigma Neural Networks

In this paper, a gradient method with momentum for sigma-pi-sigma neural networks (SPSNN) is considered in order to accelerate the convergence of the learning procedure for the network weights. The momentum coefficient is chosen in an adaptive manner, and the corresponding weak convergence and strong convergence results are proved.

CONVERGENCE OF ONLINE GRADIENT METHOD WITH A PENALTY TERM FOR FEEDFORWARD NEURAL NETWORKS WITH STOCHASTIC INPUTS

Online gradient algorithm has been widely used as a learning algorithm for feedforward neural network training. In this paper, we prove a weak convergence theorem of an online gradient algorithm with a penalty term, assuming that the training examples are input in a stochastic way. The monotonicity of the error function in the iteration and the boundedness of the weight are both guaranteed. We also present a numerical experiment to support our results.

A progressive surrogate gradient learning for memristive spiking neural network

In recent years, spiking neural networks(SNNs) have received increasing attention of research in the field of artificial intelligence due to their high biological plausibility, low energy consumption, and abundant spatio-temporal information.However, the non-differential spike activity makes SNNs more difficult to train in supervised training. Most existing methods focusing on introducing an approximated derivative to replace it, while they are often based on static surrogate functions. In this paper, we propose a progressive surrogate gradient learning for backpropagation of SNNs, which is able to approximate the step function gradually and to reduce information loss. Furthermore, memristor cross arrays are used for speeding up calculation and reducing system energy consumption for their hardware advantage. The proposed algorithm is evaluated on both static and neuromorphic datasets using fully connected and convolutional network architecture, and the experimental results indicate that our approach has a high performance compared with previous research.

Modeling and analysis of porosity and compressive strength of gradient Al_2O_3-ZrO_2 ceramic lter using BP neural network

BP neural network was used in this study to model the porosity and the compressive strength of a gradient Al2Q-ZrO2 ceramic foam filter prepared by centrifugal slip casting. The influences of the load applied on the epispastic polystyrene template （F）, the centrifugal acceleration （V） and sintering temperature （T） on the porosity （P） and compressive strength （a） of the sintered products were studied by using the registered three-layer BP model. The accuracy of the model was verified by comparing the BP model predicted results with the experimental ones. Results show that the model prediction agrees with the experimental data within a reasonable experimental error, indicating that the three-layer BP network based modeling is effective in predicting both the properties and processing parameters in designing the gradient Al203-ZrO2 ceramic foam filter. The prediction results show that the porosity percentage increases and compressive strength decreases with an increase in the applied load on epispastic polystyrene template. As for the influence of sintering temperature, the porosity percentage decreases monotonically with an increase in sintering temperature, yet the compressive strength first increases and then decreases slightly in a given temperature range. Furthermore, the porosity percentage changes little but the compressive strength first increases and then decreases when the centrifugal acceleration increases.

Rockburst Intensity Grade Prediction Model Based on Batch Gradient Descent and Multi-Scale Residual Deep Neural Network

Rockburst is a phenomenon in which free surfaces are formed during excavation,which subsequently causes the sudden release of energy in the construction of mines and tunnels.Light rockburst only peels off rock slices without ejection,while severe rockburst causes casualties and property loss.The frequency and degree of rockburst damage increases with the excavation depth.Moreover,rockburst is the leading engineering geological hazard in the excavation process,and thus the prediction of its intensity grade is of great significance to the development of geotechnical engineering.Therefore,the prediction of rockburst intensity grade is one problem that needs to be solved urgently.By comprehensively considering the occurrence mechanism of rockburst,this paper selects the stress index(σθ/σc),brittleness index(σ_(c)/σ_(t)),and rock elastic energy index(Wet)as the rockburst evaluation indexes through the Spearman coefficient method.This overcomes the low accuracy problem of a single evaluation index prediction method.Following this,the BGD-MSR-DNN rockburst intensity grade prediction model based on batch gradient descent and a multi-scale residual deep neural network is proposed.The batch gradient descent(BGD)module is used to replace the gradient descent algorithm,which effectively improves the efficiency of the network and reduces the model training time.Moreover,the multi-scale residual(MSR)module solves the problem of network degradation when there are too many hidden layers of the deep neural network(DNN),thus improving the model prediction accuracy.The experimental results reveal the BGDMSR-DNN model accuracy to reach 97.1%,outperforming other comparable models.Finally,actual projects such as Qinling Tunnel and Daxiangling Tunnel,reached an accuracy of 100%.The model can be applied in mines and tunnel engineering to realize the accurate and rapid prediction of rockburst intensity grade.

GNNSched:面向GPU的图神经网络推理任务调度框架

由于频繁的显存访问,图神经网络GNN在GPU上运行时往往资源利用率较低。现有的推理框架由于没有考虑GNN输入的不规则性,直接适用到GNN进行推理任务共置时可能会超出显存容量导致任务失败。对于GNN推理任务,需要根据其输入特点预先分析并发任务的显存占用情况,以确保并发任务在GPU上的成功共置。此外,多租户场景提交的推理任务亟需灵活的调度策略,以满足并发推理任务的服务质量要求。为了解决上述问题,提出了GNNSched,其在GPU上高效管理GNN推理任务的共置运行。具体来说,GNNSched将并发推理任务组织为队列,并在算子粒度上根据成本函数估算每个任务的显存占用情况。GNNSched实现了多种调度策略来生成任务组,这些任务组被迭代地提交到GPU并发执行。实验结果表明,GNNSched能够满足并发GNN推理任务的服务质量并降低推理任务的响应时延。

Anti-noise performance of the pulse coupled neural network applied in discrimination of neutron and gamma-ray

In this study,the anti-noise performance of a pulse-coupled neural network(PCNN)was investigated in the neutron and gamma-ray(n-γ)discrimination field.The experiments were conducted in two groups.In the first group,radiation pulse signals were pre-processed using a Fourier filter to reduce the original noise in the signals,whereas in the second group,the original noise was left untouched to simulate an extremely high-noise scenario.For each part,artificial Gaussian noise with different intensity levels was added to the signals prior to the discrimination process.In the aforementioned conditions,the performance of the PCNN was evaluated and compared with five other commonly used methods of n-γdiscrimination:(1)zero crossing,(2)charge comparison,(3)vector projection,(4)falling edge percentage slope,and(5)frequency gradient analysis.The experimental results showed that the PCNN method significantly outperforms other methods with outstanding FoM-value at all noise levels.Furthermore,the fluctuations in FoM-value of PCNN were significantly better than those obtained via other methods at most noise levels and only slightly worse than those obtained via the charge comparison and zerocrossing methods under extreme noise conditions.Additionally,the changing patterns and fluctuations of the FoMvalue were evaluated under different noise conditions.Hence,based on the results,the parameter selection strategy of the PCNN was presented.In conclusion,the PCNN method is suitable for use in high-noise application scenarios for n-γdiscrimination because of its stability and remarkable discrimination performance.It does not rely on strict parameter settings and can realize satisfactory performance over a wide parameter range.

Indian stock market prediction using artificial neural networks on tick data

Introduction:Nowadays,the most significant challenges in the stock market is to predict the stock prices.The stock price data represents a financial time series data which becomes more difficult to predict due to its characteristics and dynamic nature.Case description:Support Vector Machines(SVM)and Artificial Neural Networks(ANN)are widely used for prediction of stock prices and its movements.Every algorithm has its way of learning patterns and then predicting.Artificial Neural Network(ANN)is a popular method which also incorporate technical analysis for making predictions in financial markets.Discussion and evaluation:Most common techniques used in the forecasting of financial time series are Support Vector Machine(SVM),Support Vector Regression(SVR)and Back Propagation Neural Network(BPNN).In this article,we use neural networks based on three different learning algorithms,i.e.,Levenberg-Marquardt,Scaled Conjugate Gradient and Bayesian Regularization for stock market prediction based on tick data as well as 15-min data of an Indian company and their results compared.Conclusion:All three algorithms provide an accuracy of 99.9%using tick data.The accuracy over 15-min dataset drops to 96.2%,97.0%and 98.9%for LM,SCG and Bayesian Regularization respectively which is significantly poor in comparison with that of results obtained using tick data.

Modeling and optimum operating conditions for FCCU using artificial neural network

A self-organizing radial basis function(RBF) neural network(SODM-RBFNN) was presented for predicting the production yields and operating optimization. Gradient descent algorithm was used to optimize the widths of RBF neural network with the initial parameters obtained by k-means learning method. During the iteration procedure of the algorithm, the centers of the neural network were optimized by using the gradient method with these optimized width values. The computational efficiency was maintained by using the multi-threading technique. SODM-RBFNN consists of two RBF neural network models: one is a running model used to predict the product yields of fluid catalytic cracking unit(FCCU) and optimize its operating parameters; the other is a learning model applied to construct or correct a RBF neural network. The running model can be updated by the learning model according to an accuracy criterion. The simulation results of a five-lump kinetic model exhibit its accuracy and generalization capabilities, and practical application in FCCU illustrates its effectiveness.

Application of New Type BP Neural Networks for Magnetic Measurement

Magnetic measurement is a typical inverse problem in Biomedical field. In this kind of problem we always need to locate the positions and moments of one or more magnetic dipoles. Although using the traditional methods to solve this kind of inverse problem has all kinds of shortcomings, BPNN (Back Propagation Neural Networks) method can be used to solve this typical inverse problem fast enough for real time measurement. In the traditional BPNN method, gradient descent search method is performed for error propagation. In this paper the authors propose a new algorithm that Newton method is performed for error propagation. For the cost function is highly nonconvex in the magnetic measurement problem, the new kind of BPNN can get convergent results quickly and precisely. A simulation result for this method is also presented.

基于强化联邦GNN的个性化公共安全突发事件检测

近年来,将公共安全数据转换为图的形式,通过图神经网络(GNN)构造节点表示应用于下游任务的方法,充分利用了公共安全数据的实体与关联信息,取得了较好的效果.为了提高模型的有效性,需要大量的高质量数据,但是高质量的数据通常归属于政府、公司和组织,很难通过数据集中的方式使模型学习到有效的事件检测模型.由于各数据拥有方的关注主题与收集时间不同,数据之间存在Non-IID的问题.传统的假设一个全局模型可以适合所有客户端的方法难以解决此类问题.提出了基于强化联邦图神经网络的个性化公共安全突发事件检测方法PPSED,各客户端采用多方协作的方式训练个性化的模型来解决本地的突发事件检测任务.设计了联邦公共安全突发事件检测模型的本地训练与梯度量化模块,采用基于图采样的minibatch机制的GraphSage构造公共安全突发事件检测本地模型,以减小数据Non-IID的影响,采用梯度量化方法减小梯度通信的消耗.设计了基于随机图嵌入的客户端状态感知模块,在保护隐私的同时,更好地保留客户端模型有价值的梯度信息.设计了强化联邦图神经网络的个性化梯度聚合与量化策略,采用DDPG拟合个性化联邦学习梯度聚合加权策略,并根据权重决定是否对梯度进行量化,对模型的性能与通信压力进行平衡.通过在微博平台收集的公共安全数据集和3个公开的图数据集进行了大量的实验,实验结果表明了所提方法的有效性.

On fine-grained visual explanation in convolutional neural networks

Existing explanation methods for Convolutional Neural Networks(CNNs)lack the pixel-level visualization explanations to generate the reliable fine-grained decision features.Since there are inconsistencies between the explanation and the actual behavior of the model to be interpreted,we propose a Fine-Grained Visual Explanation for CNN,namely F-GVE,which produces a fine-grained explanation with higher consistency to the decision of the original model.The exact backward class-specific gradients with respect to the input image is obtained to highlight the object-related pixels the model used to make prediction.In addition,for better visualization and less noise,F-GVE selects an appropriate threshold to filter the gradient during the calculation and the explanation map is obtained by element-wise multiplying the gradient and the input image to show fine-grained classification decision features.Experimental results demonstrate that F-GVE has good visual performances and highlights the importance of fine-grained decision features.Moreover,the faithfulness of the explanation in this paper is high and it is effective and practical on troubleshooting and debugging detection.

Performance comparison of training algorithms for the estimation of B?hme abrasion resistance using neural networks

Natural stones used as floor and wall coverings are exposed to many different abrasive forces,so it is essential to choose suitable materials for wear resistance in terms of the life of the structure.The abrasion resistance of natural stones can be determined in the laboratory by applying the B?hme abrasion resistance(BAR)test.However,the direct analysis of BAR in the laboratory has disadvantages such as wasting time and energy,experimental errors,and health impacts.To eliminate these disadvantages,the estimation of BAR using artificial neural networks(ANN)was proposed.Different natural stone samples were collected from Türkiye,and uniaxial compressive strength(UCS),flexural strength(FS),water absorption rate(WA),unit volume weight(UW),effective porosity(n),and BAR tests were carried out.The outputs of these tests were gathered and a data set,consisting of a total of 105 data,was randomly divided into two groups:testing and training.In the current study,the success of three different training algorithms of Levenberg-Marquardt(LM),Bayesian regularization(BR),and scaled conjugate gradient(SCG)were compared for BAR prediction of natural stones.Statistical criteria such as coefficient of determination(R~2),mean square error(MSE),mean square error(RMSE),and mean absolute percentage error(MAPE),which are widely used and adopted in the literature,were used to determine predictive validity.The findings of the study indicated that ANN is a valid method for estimating the BAR value.Also,the LM algorithm(R~2=0.9999,MSE=0.0001,RMSE=0.0110,and MAPE=0.0487)in training and the BR algorithm(R~2=0.9896,MSE=0.0589,RMSE=0.2427,and MAPE=1.2327)in testing showed the best prediction performance.It has been observed that the proposed method is quite practical to implement.Using the artificial neural networks method will provide an advantage in similar laborintensive experimental studies.

EFFICIENT GRADIENT DESCENT METHOD OFRBF NEURAL ENTWORKS WITHADAPTIVE LEARNING RATE

A new algorithm to exploit the learning rates of gradient descent method is presented, based on the second-order Taylor expansion of the error energy function with respect to learning rate, at some values decided by "award-punish" strategy. Detailed deduction of the algorithm applied to RBF networks is given. Simulation studies show that this algorithm can increase the rate of convergence and improve the performance of the gradient descent method.

Fingerprint Identification by Artificial Neural Network

Biometric techniques require critical operations of digital processing for identification of individuals. In this context, this paper aims to develop a system for automatic processing of fingerprint identification by their minutiae using Artificial Neural Networks （ANN）, which reveals to be highly effective. The ANN method implemented is a based on Multi-Layer Perceptron （MLP） model, which utilizes the algorithm of retro-propagation of gradient during the learning process. In such a process, the mean square error generated represents the specific parameter for the identification phase by comparing a fingerprint taken from a crime scene with those of a reference database.