Scale adaptive fitness evaluation‐based particle swarm optimisation for hyperparameter and architecture optimisation in neural networks and deep learning

Research into automatically searching for an optimal neural network(NN)by optimi-sation algorithms is a significant research topic in deep learning and artificial intelligence.However,this is still challenging due to two issues:Both the hyperparameter and ar-chitecture should be optimised and the optimisation process is computationally expen-sive.To tackle these two issues,this paper focusses on solving the hyperparameter and architecture optimization problem for the NN and proposes a novel light‐weight scale‐adaptive fitness evaluation‐based particle swarm optimisation(SAFE‐PSO)approach.Firstly,the SAFE‐PSO algorithm considers the hyperparameters and architectures together in the optimisation problem and therefore can find their optimal combination for the globally best NN.Secondly,the computational cost can be reduced by using multi‐scale accuracy evaluation methods to evaluate candidates.Thirdly,a stagnation‐based switch strategy is proposed to adaptively switch different evaluation methods to better balance the search performance and computational cost.The SAFE‐PSO algorithm is tested on two widely used datasets:The 10‐category(i.e.,CIFAR10)and the 100−cate-gory(i.e.,CIFAR100).The experimental results show that SAFE‐PSO is very effective and efficient,which can not only find a promising NN automatically but also find a better NN than compared algorithms at the same computational cost.

Neural Network-Based Limiter with Transfer Learning

Recent works have shown that neural networks are promising parameter-free limiters for a variety of numerical schemes(Morgan et al.in A machine learning approach for detect-ing shocks with high-order hydrodynamic methods.et al.in J Comput Phys 367:166-191.,2018;Veiga et al.in European Conference on Computational Mechanics andⅦEuropean Conference on Computational Fluid Dynamics,vol.1,pp.2525-2550.ECCM.,2018).Following this trend,we train a neural network to serve as a shock-indicator function using simulation data from a Runge-Kutta discontinuous Galer-kin(RKDG)method and a modal high-order limiter(Krivodonova in J Comput Phys 226:879-896.,2007).With this methodology,we obtain one-and two-dimensional black-box shock-indicators which are then coupled to a standard limiter.Furthermore,we describe a strategy to transfer the shock-indicator to a residual distribution(RD)scheme without the need for a full training cycle and large data-set,by finding a mapping between the solution feature spaces from an RD scheme to an RKDG scheme,both in one-and two-dimensional problems,and on Cartesian and unstruc-tured meshes.We report on the quality of the numerical solutions when using the neural network shock-indicator coupled to a limiter,comparing its performance to traditional lim-iters,for both RKDG and RD schemes.

Adaptive Learning with Large Variability of Teaching Signals for Neural Networks and Its Application to Motion Control of an Industrial Robot

Recently, various control methods represented by proportional-integral-derivative （PID） control are used for robotic control. To cope with the requirements for high response and precision, advanced feedforward controllers such as gravity compensator, Coriolis/centrifugal force compensator and friction compensators have been built in the controller. Generally, it causes heavy computational load when calculating the compensating value within a short sampling period. In this paper, integrated recurrent neural networks are applied as a feedforward controller for PUMA560 manipulator. The feedforward controller works instead of gravity and Coriolis/centrifugal force compensators. In the learning process of the neural network by using back propagation algorithm, the learning coefficient and gain of sigmoid function are tuned intuitively and empirically according to teaching signals. The tuning is complicated because it is being conducted by trial and error. Especially, when the scale of teaching signal is large, the problem becomes crucial. To cope with the problem which concerns the learning performance, a simple and adaptive learning technique for large scale teaching signals is proposed. The learning techniques and control effectiveness are evaluated through simulations using the dynamic model of PUMA560 manipulator.

Adaptive Iterative Learning Control for Nonlinear Time-delay Systems with Periodic Disturbances Using FSE-neural Network

An adaptive iterative learning control scheme is presented for a class of strict-feedback nonlinear time-delay systems, with unknown nonlinearly parameterised and time-varying disturbed functions of known periods. Radial basis function neural network and Fourier series expansion （FSE） are combined into a new function approximator to model each suitable disturbed function in systems. The requirement of the traditional iterative learning control algorithm on the nonlinear functions （such as global Lipschitz condition） is relaxed. Furthermore, by using appropriate Lyapunov-Krasovskii functionals, all signs in the closed loop system are guaranteed to be semiglobally uniformly ultimately bounded, and the output of the system is proved to converge to the desired trajectory. A simulation example is provided to illustrate the effectiveness of the control scheme.

Hyperparameter Tuning for Deep Neural Networks Based Optimization Algorithm

For training the present Neural Network(NN)models,the standard technique is to utilize decaying Learning Rates(LR).While the majority of these techniques commence with a large LR,they will decay multiple times over time.Decaying has been proved to enhance generalization as well as optimization.Other parameters,such as the network’s size,the number of hidden layers,drop-outs to avoid overfitting,batch size,and so on,are solely based on heuristics.This work has proposed Adaptive Teaching Learning Based(ATLB)Heuristic to identify the optimal hyperparameters for diverse networks.Here we consider three architec-tures Recurrent Neural Networks(RNN),Long Short Term Memory(LSTM),Bidirectional Long Short Term Memory(BiLSTM)of Deep Neural Networks for classification.The evaluation of the proposed ATLB is done through the various learning rate schedulers Cyclical Learning Rate(CLR),Hyperbolic Tangent Decay(HTD),and Toggle between Hyperbolic Tangent Decay and Triangular mode with Restarts(T-HTR)techniques.Experimental results have shown the performance improvement on the 20Newsgroup,Reuters Newswire and IMDB dataset.

Adaptive Momentum-Backpropagation Algorithm for Flood Prediction and Management in the Internet of Things

Flooding is a hazardous natural calamity that causes significant damage to lives and infrastructure in the real world.Therefore,timely and accurate decision-making is essential for mitigating flood-related damages.The traditional flood prediction techniques often encounter challenges in accuracy,timeliness,complexity in handling dynamic flood patterns and leading to substandard flood management strategies.To address these challenges,there is a need for advanced machine learning models that can effectively analyze Internet of Things(IoT)-generated flood data and provide timely and accurate flood predictions.This paper proposes a novel approach-the Adaptive Momentum and Backpropagation(AM-BP)algorithm-for flood prediction and management in IoT networks.The AM-BP model combines the advantages of an adaptive momentum technique with the backpropagation algorithm to enhance flood prediction accuracy and efficiency.Real-world flood data is used for validation,demonstrating the superior performance of the AM-BP algorithm compared to traditional methods.In addition,multilayer high-end computing architecture(MLCA)is used to handle weather data such as rainfall,river water level,soil moisture,etc.The AM-BP’s real-time abilities enable proactive flood management,facilitating timely responses and effective disaster mitigation.Furthermore,the AM-BP algorithm can analyze large and complex datasets,integrating environmental and climatic factors for more accurate flood prediction.The evaluation result shows that the AM-BP algorithm outperforms traditional approaches with an accuracy rate of 96%,96.4%F1-Measure,97%Precision,and 95.9%Recall.The proposed AM-BP model presents a promising solution for flood prediction and management in IoT networks,contributing to more resilient and efficient flood control strategies,and ensuring the safety and well-being of communities at risk of flooding.

An Intelligent Neural Networks System for Adaptive Learning and Prediction of a Bioreactor Benchmark Process

适应学习和一个高度非线性、变化时间的生物反应器基准过程的预言用神经原线，为在 G2 实时聪明的环境开发并且部署神经网络的一个图形的工具箱，和新修改 Broyden 被学习，弗莱彻， Goldfarb ，并且 Shanno ( BFGS )伪--牛顿算法。为神经网络被开发并且由介绍学习率到完整的存储器 BFGS 算法的一个新搜索方法嵌进神经原线的背繁殖(BP ) 的适应学习的修改 BFGS 算法。模拟结果证明适应学习和神经网络系统能快速追踪的预言生物反应器的变化时间、非线性的行为。

Adaptive learning with guaranteed stability for discrete-time recurrent neural networks

To avoid unstable learning, a stable adaptive learning algorithm was proposed for discrete-time recurrent neural networks. Unlike the dynamic gradient methods, such as the backpropagation through time and the real time recurrent learning, the weights of the recurrent neural networks were updated online in terms of Lyapunov stability theory in the proposed learning algorithm, so the learning stability was guaranteed. With the inversion of the activation function of the recurrent neural networks, the proposed learning algorithm can be easily implemented for solving varying nonlinear adaptive learning problems and fast convergence of the adaptive learning process can be achieved. Simulation experiments in pattern recognition show that only 5 iterations are needed for the storage of a 15×15 binary image pattern and only 9 iterations are needed for the perfect realization of an analog vector by an equilibrium state with the proposed learning algorithm.

Investigation on the Chinese Text Sentiment Analysis Based on Convolutional Neural Networks in Deep Learning

Nowadays,the amount of wed data is increasing at a rapid speed,which presents a serious challenge to the web monitoring.Text sentiment analysis,an important research topic in the area of natural language processing,is a crucial task in the web monitoring area.The accuracy of traditional text sentiment analysis methods might be degraded in dealing with mass data.Deep learning is a hot research topic of the artificial intelligence in the recent years.By now,several research groups have studied the sentiment analysis of English texts using deep learning methods.In contrary,relatively few works have so far considered the Chinese text sentiment analysis toward this direction.In this paper,a method for analyzing the Chinese text sentiment is proposed based on the convolutional neural network(CNN)in deep learning in order to improve the analysis accuracy.The feature values of the CNN after the training process are nonuniformly distributed.In order to overcome this problem,a method for normalizing the feature values is proposed.Moreover,the dimensions of the text features are optimized through simulations.Finally,a method for updating the learning rate in the training process of the CNN is presented in order to achieve better performances.Experiment results on the typical datasets indicate that the accuracy of the proposed method can be improved compared with that of the traditional supervised machine learning methods,e.g.,the support vector machine method.

The adaptive control using BP neural networks for a nonlinear servo-motor

The servo-motor possesses a strongly nonlinear property due to the effect of the stimulating input voltage, load-torque and environmental operating conditions. So it is rather difficult to derive a traditional mathematical model which is capable of expressing both its dynamics and steady-state characteristics. A neural network-based adaptive control strategy is proposed in this paper. In this method, two neural networks have been adopted for system identification （NNI） and control （NNC）, respectively. Then, the commonly-used specialized learning has been modified, by taking the NNI output as the approximation output of the servo-motor during the weights training to get sensitivity information. Moreover, the rule for choosing the learning rate is given on the basis of the analysis of Lyapunov stability. Finally, an example of applying the proposed control strategy on a servo-motor is presented to show its effectiveness.

Complex-Valued Neural Networks:A Comprehensive Survey

Complex-valued neural networks(CVNNs)have shown their excellent efficiency compared to their real counterparts in speech enhancement,image and signal processing.Researchers throughout the years have made many efforts to improve the learning algorithms and activation functions of CVNNs.Since CVNNs have proven to have better performance in handling the naturally complex-valued data and signals,this area of study will grow and expect the arrival of some effective improvements in the future.Therefore,there exists an obvious reason to provide a comprehensive survey paper that systematically collects and categorizes the advancement of CVNNs.In this paper,we discuss and summarize the recent advances based on their learning algorithms,activation functions,which is the most challenging part of building a CVNN,and applications.Besides,we outline the structure and applications of complex-valued convolutional,residual and recurrent neural networks.Finally,we also present some challenges and future research directions to facilitate the exploration of the ability of CVNNs.

Fetal ECG Extraction Based on Adaptive Linear Neural Network

Fetal ECG extraction has the vital significance for fetal monitoring.This paper introduces a method of extracting fetal ECG based on adaptive linear neural network.The method can be realized by training a small quantity of data.In addition,a better result can be achieved by improving neural network structure.Thus,more easily identified fetal ECG can be extracted.Experimental results show that the adaptive linear neural network can be used to extract fetal ECG from maternal abdominal signal effectively.What's more,a clearer fetal ECG can be extracted by improving neural network structure.

Neural networks-based iterative learning control consensus for periodically time-varying multi-agent systems

In this paper,the problem of adaptive iterative learning based consensus control for periodically time-varying multi-agent systems is studied,in which the dynamics of each follower are driven by nonlinearly parameterized terms with periodic disturbances.Neural networks and Fourier base expansions are introduced to describe the periodically time-varying dynamic terms.On this basis,an adaptive learning parameter with a positively convergent series term is constructed,and a distributed control protocol based on local signals between agents is designed to ensure accurate consensus of the closed-loop systems.Furthermore,consensus algorithm is generalized to solve the formation control problem.Finally,simulation experiments are implemented through MATLAB to demonstrate the effectiveness of the method used.

Corporate Credit Ratings Based on Hierarchical Heterogeneous Graph Neural Networks

order to help investors understand the credit status of target corporations and reduce investment risks,the corporate credit rating model has become an important evaluation tool in the financial market.These models are based on statistical learning,machine learning and deep learning especially graph neural networks(GNNs).However,we found that only few models take the hierarchy,heterogeneity or unlabeled data into account in the actual corporate credit rating process.Therefore,we propose a novel framework named hierarchical heterogeneous graph neural networks(HHGNN),which can fully model the hierarchy of corporate features and the heterogeneity of relationships between corporations.In addition,we design an adversarial learning block to make full use of the rich unlabeled samples in the financial data.Extensive experiments conducted on the public-listed corporate rating dataset prove that HHGNN achieves SOTA compared to the baseline methods.

Artificial Neural Network Methods for the Solution of Second Order Boundary Value Problems

We present a method for solving partial differential equations using artificial neural networks and an adaptive collocation strategy.In this procedure,a coarse grid of training points is used at the initial training stages,while more points are added at later stages based on the value of the residual at a larger set of evaluation points.This method increases the robustness of the neural network approximation and can result in significant computational savings,particularly when the solution is non-smooth.Numerical results are presented for benchmark problems for scalar-valued PDEs,namely Poisson and Helmholtz equations,as well as for an inverse acoustics problem.

Enhanced Marathi Speech Recognition Facilitated by Grasshopper Optimisation-Based Recurrent Neural Network

Communication is a significant part of being human and living in the world.Diverse kinds of languages and their variations are there;thus,one person can speak any language and cannot effectively communicate with one who speaks that language in a different accent.Numerous application fields such as education,mobility,smart systems,security,and health care systems utilize the speech or voice recognition models abundantly.Though,various studies are focused on the Arabic or Asian and English languages by ignoring other significant languages like Marathi that leads to the broader research motivations in regional languages.It is necessary to understand the speech recognition field,in which the major concentrated stages are feature extraction and classification.This paper emphasis developing a Speech Recognition model for the Marathi language by optimizing Recurrent Neural Network(RNN).Here,the preprocessing of the input signal is performed by smoothing and median filtering.After preprocessing the feature extraction is carried out using MFCC and Spectral features to get precise features from the input Marathi Speech corpus.The optimized RNN classifier is used for speech recognition after completing the feature extraction task,where the optimization of hidden neurons in RNN is performed by the Grasshopper Optimization Algorithm(GOA).Finally,the comparison with the conventional techniques has shown that the proposed model outperforms most competing models on a benchmark dataset.

Application of Artificial Neural Network in Robotic Hybrid Position/Force Control

A hybrid position/force controller is designed for the joint 2 and the joint 3 of thePUMA 560 robot.The hybrid controller includes a multilayered neural network,which canidentify the dynamics of the contacted environment and can optimize the parameters of PIDcontroller.The experimental results show that after having been trained,the robot has sta-ble response to the training patterns and strong adaptive ability to the situation between thepatterns.

Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Neural Network State Learning Based Adaptive Terminal ILC for Tracking Iteration-varying Target Points

Terminal iterative learning control(TILC) is developed to reduce the error between system output and a fixed desired point at the terminal end of operation interval over iterations under strictly identical initial conditions. In this work, the initial states are not required to be identical further but can be varying from iteration to iteration. In addition, the desired terminal point is not fixed any more but is allowed to change run-to-run. Consequently, a new adaptive TILC is proposed with a neural network initial state learning mechanism to achieve the learning objective over iterations. The neural network is used to approximate the effect of iteration-varying initial states on the terminal output and the neural network weights are identified iteratively along the iteration axis.A dead-zone scheme is developed such that both learning and adaptation are performed only if the terminal tracking error is outside a designated error bound. It is shown that the proposed approach is able to track run-varying terminal desired points fast with a specified tracking accuracy beyond the initial state variance.

Assessment of dairy cow feed intake based on BP neural network with polynomial decay learning rate

To overcome the shortcomings of traditional dairy cow feed intake assessment model andBP neural network, this paper proposes a method of optimizing BP neural network usingpolynomial decay learning rate, taking the cow’s body weight, lying duration, lying times,walking steps, foraging duration and concentrate-roughage ratio as input variables andtaking the actual feed intake is the output variable to establish a dairy cow feed intakeassessment model, and the model is trained and verified by experimental data collectedon site. For the sake of comparative study, feed intake is simultaneously assessed by SVRmodel, KNN logistic regression model, traditional BP neural network model, and multilayerBP neural network model. The results show that the established BP model using the polynomial decay learning rate has the highest assessment accuracy, the MSPE, RMSE, MAE,MAPE and R2 are 0.043 kg2/d and 0.208 kg/d, 0.173 kg/d, 1.37% and 0.94 respectively. Compared with SVR model and KNN mode, the RMSE value reduced by 43.9% and 26.5%, it isalso found that the model designed in this paper has many advantages in comparison withthe BP model and multilayer BP model in terms of precision and generalization. Therefore,this method is ready to be applied for accurately evaluating the dairy cow feed intake, andit can provide theoretical guidance and technical support for the precise-feeding and canalso be of high significance in the improvement of dairy precise-breeding.