Application of Random Search Methods in the Determination of Learning Rate for Training Container Dwell Time Data Using Artificial Neural Networks

Purpose: This study aimed to enhance the prediction of container dwell time, a crucial factor for optimizing port operations, resource allocation, and supply chain efficiency. Determining an optimal learning rate for training Artificial Neural Networks (ANNs) has remained a challenging task due to the diverse sizes, complexity, and types of data involved. Design/Method/Approach: This research used a RandomizedSearchCV algorithm, a random search approach, to bridge this knowledge gap. The algorithm was applied to container dwell time data from the TOS system of the Port of Tema, which included 307,594 container records from 2014 to 2022. Findings: The RandomizedSearchCV method outperformed standard training methods both in terms of reducing training time and improving prediction accuracy, highlighting the significant role of the constant learning rate as a hyperparameter. Research Limitations and Implications: Although the study provides promising outcomes, the results are limited to the data extracted from the Port of Tema and may differ in other contexts. Further research is needed to generalize these findings across various port systems. Originality/Value: This research underscores the potential of RandomizedSearchCV as a valuable tool for optimizing ANN training in container dwell time prediction. It also accentuates the significance of automated learning rate selection, offering novel insights into the optimization of container dwell time prediction, with implications for improving port efficiency and supply chain operations.

Spectral transfer-learning-based metasurface design assisted by complex-valued deep neural network

Recently,deep learning has been used to establish the nonlinear and nonintuitive mapping between physical structures and electromagnetic responses of meta-atoms for higher computational efficiency.However,to obtain sufficiently accurate predictions,the conventional deep-learning-based method consumes excessive time to collect the data set,thus hindering its wide application in this interdisciplinary field.We introduce a spectral transfer-learning-based metasurface design method to achieve excellent performance on a small data set with only 1000 samples in the target waveband by utilizing open-source data from another spectral range.We demonstrate three transfer strategies and experimentally quantify their performance,among which the“frozen-none”robustly improves the prediction accuracy by∼26%compared to direct learning.We propose to use a complex-valued deep neural network during the training process to further improve the spectral predicting precision by∼30%compared to its real-valued counterparts.We design several typical teraherz metadevices by employing a hybrid inverse model consolidating this trained target network and a global optimization algorithm.The simulated results successfully validate the capability of our approach.Our work provides a universal methodology for efficient and accurate metasurface design in arbitrary wavebands,which will pave the way toward the automated and mass production of metasurfaces.

Fast Learning in Spiking Neural Networks by Learning Rate Adaptation

For accelerating the supervised learning by the SpikeProp algorithm with the temporal coding paradigm in spiking neural networks (SNNs), three learning rate adaptation methods (heuristic rule, delta-delta rule, and delta-bar-delta rule), which are used to speed up training in artificial neural networks, are used to develop the training algorithms for feedforward SNN. The performance of these algorithms is investigated by four experiments: classical XOR (exclusive or) problem, Iris dataset, fault diagnosis in the Tennessee Eastman process, and Poisson trains of discrete spikes. The results demonstrate that all the three learning rate adaptation methods are able to speed up convergence of SNN compared with the original SpikeProp algorithm. Furthermore, if the adaptive learning rate is used in combination with the momentum term, the two modifications will balance each other in a beneficial way to accomplish rapid and steady convergence. In the three learning rate adaptation methods, delta-bar-delta rule performs the best. The delta-bar-delta method with momentum has the fastest convergence rate, the greatest stability of training process, and the maximum accuracy of network learning. The proposed algorithms in this paper are simple and efficient, and consequently valuable for practical applications of SNN.

An Intelligent Neural Networks System for Adaptive Learning and Prediction of a Bioreactor Benchmark Process

The adaptive learning and prediction of a highly nonlinear and time-varying bioreactor benchmark process is studied using Neur-On-Line, a graphical tool kit for developing and deploying neural networks in the G2 real time intelligent environment,and a new modified Broyden, Fletcher, Goldfarb, and Shanno （BFGS） quasi-Newton algorithm. The modified BFGS algorithm for the adaptive learning of back propagation （BP） neural networks is developed and embedded into NeurOn-Line by introducing a new search method of learning rate to the full memory BFGS algorithm. Simulation results show that the adaptive learning and prediction neural network system can quicklv track the time-varving and nonlinear behavior of the bioreactor.

Adaptive Learning with Large Variability of Teaching Signals for Neural Networks and Its Application to Motion Control of an Industrial Robot

Recently, various control methods represented by proportional-integral-derivative （PID） control are used for robotic control. To cope with the requirements for high response and precision, advanced feedforward controllers such as gravity compensator, Coriolis/centrifugal force compensator and friction compensators have been built in the controller. Generally, it causes heavy computational load when calculating the compensating value within a short sampling period. In this paper, integrated recurrent neural networks are applied as a feedforward controller for PUMA560 manipulator. The feedforward controller works instead of gravity and Coriolis/centrifugal force compensators. In the learning process of the neural network by using back propagation algorithm, the learning coefficient and gain of sigmoid function are tuned intuitively and empirically according to teaching signals. The tuning is complicated because it is being conducted by trial and error. Especially, when the scale of teaching signal is large, the problem becomes crucial. To cope with the problem which concerns the learning performance, a simple and adaptive learning technique for large scale teaching signals is proposed. The learning techniques and control effectiveness are evaluated through simulations using the dynamic model of PUMA560 manipulator.

Scale adaptive fitness evaluation‐based particle swarm optimisation for hyperparameter and architecture optimisation in neural networks and deep learning

Research into automatically searching for an optimal neural network(NN)by optimi-sation algorithms is a significant research topic in deep learning and artificial intelligence.However,this is still challenging due to two issues:Both the hyperparameter and ar-chitecture should be optimised and the optimisation process is computationally expen-sive.To tackle these two issues,this paper focusses on solving the hyperparameter and architecture optimization problem for the NN and proposes a novel light‐weight scale‐adaptive fitness evaluation‐based particle swarm optimisation(SAFE‐PSO)approach.Firstly,the SAFE‐PSO algorithm considers the hyperparameters and architectures together in the optimisation problem and therefore can find their optimal combination for the globally best NN.Secondly,the computational cost can be reduced by using multi‐scale accuracy evaluation methods to evaluate candidates.Thirdly,a stagnation‐based switch strategy is proposed to adaptively switch different evaluation methods to better balance the search performance and computational cost.The SAFE‐PSO algorithm is tested on two widely used datasets:The 10‐category(i.e.,CIFAR10)and the 100−cate-gory(i.e.,CIFAR100).The experimental results show that SAFE‐PSO is very effective and efficient,which can not only find a promising NN automatically but also find a better NN than compared algorithms at the same computational cost.

Adaptive learning with guaranteed stability for discrete-time recurrent neural networks

To avoid unstable learning, a stable adaptive learning algorithm was proposed for discrete-time recurrent neural networks. Unlike the dynamic gradient methods, such as the backpropagation through time and the real time recurrent learning, the weights of the recurrent neural networks were updated online in terms of Lyapunov stability theory in the proposed learning algorithm, so the learning stability was guaranteed. With the inversion of the activation function of the recurrent neural networks, the proposed learning algorithm can be easily implemented for solving varying nonlinear adaptive learning problems and fast convergence of the adaptive learning process can be achieved. Simulation experiments in pattern recognition show that only 5 iterations are needed for the storage of a 15×15 binary image pattern and only 9 iterations are needed for the perfect realization of an analog vector by an equilibrium state with the proposed learning algorithm.

Adaptive Iterative Learning Control for Nonlinear Time-delay Systems with Periodic Disturbances Using FSE-neural Network

An adaptive iterative learning control scheme is presented for a class of strict-feedback nonlinear time-delay systems, with unknown nonlinearly parameterised and time-varying disturbed functions of known periods. Radial basis function neural network and Fourier series expansion （FSE） are combined into a new function approximator to model each suitable disturbed function in systems. The requirement of the traditional iterative learning control algorithm on the nonlinear functions （such as global Lipschitz condition） is relaxed. Furthermore, by using appropriate Lyapunov-Krasovskii functionals, all signs in the closed loop system are guaranteed to be semiglobally uniformly ultimately bounded, and the output of the system is proved to converge to the desired trajectory. A simulation example is provided to illustrate the effectiveness of the control scheme.

Investigation on the Chinese Text Sentiment Analysis Based on Convolutional Neural Networks in Deep Learning

Nowadays,the amount of wed data is increasing at a rapid speed,which presents a serious challenge to the web monitoring.Text sentiment analysis,an important research topic in the area of natural language processing,is a crucial task in the web monitoring area.The accuracy of traditional text sentiment analysis methods might be degraded in dealing with mass data.Deep learning is a hot research topic of the artificial intelligence in the recent years.By now,several research groups have studied the sentiment analysis of English texts using deep learning methods.In contrary,relatively few works have so far considered the Chinese text sentiment analysis toward this direction.In this paper,a method for analyzing the Chinese text sentiment is proposed based on the convolutional neural network(CNN)in deep learning in order to improve the analysis accuracy.The feature values of the CNN after the training process are nonuniformly distributed.In order to overcome this problem,a method for normalizing the feature values is proposed.Moreover,the dimensions of the text features are optimized through simulations.Finally,a method for updating the learning rate in the training process of the CNN is presented in order to achieve better performances.Experiment results on the typical datasets indicate that the accuracy of the proposed method can be improved compared with that of the traditional supervised machine learning methods,e.g.,the support vector machine method.

The adaptive control using BP neural networks for a nonlinear servo-motor

The servo-motor possesses a strongly nonlinear property due to the effect of the stimulating input voltage, load-torque and environmental operating conditions. So it is rather difficult to derive a traditional mathematical model which is capable of expressing both its dynamics and steady-state characteristics. A neural network-based adaptive control strategy is proposed in this paper. In this method, two neural networks have been adopted for system identification （NNI） and control （NNC）, respectively. Then, the commonly-used specialized learning has been modified, by taking the NNI output as the approximation output of the servo-motor during the weights training to get sensitivity information. Moreover, the rule for choosing the learning rate is given on the basis of the analysis of Lyapunov stability. Finally, an example of applying the proposed control strategy on a servo-motor is presented to show its effectiveness.

Fetal ECG Extraction Based on Adaptive Linear Neural Network

Fetal ECG extraction has the vital significance for fetal monitoring.This paper introduces a method of extracting fetal ECG based on adaptive linear neural network.The method can be realized by training a small quantity of data.In addition,a better result can be achieved by improving neural network structure.Thus,more easily identified fetal ECG can be extracted.Experimental results show that the adaptive linear neural network can be used to extract fetal ECG from maternal abdominal signal effectively.What's more,a clearer fetal ECG can be extracted by improving neural network structure.

Identification of dynamic systems using support vector regression neural networks

A novel adaptive support vector regression neural network （SVR-NN） is proposed, which combines respectively merits of support vector machines and a neural network. First, a support vector regression approach is applied to determine the initial structure and initial weights of the SVR-NN so that the network architecture is easily determined and the hidden nodes can adaptively be constructed based on support vectors. Furthermore, an annealing robust learning algorithm is presented to adjust these hidden node parameters as well as the weights of the SVR-NN. To test the validity of the proposed method, it is demonstrated that the adaptive SVR-NN can be used effectively for the identification of nonlinear dynamic systems. Simulation results show that the identification schemes based on the SVR-NN give considerably better performance and show faster learning in comparison to the previous neural network method.

Complex-Valued Neural Networks:A Comprehensive Survey

Complex-valued neural networks(CVNNs)have shown their excellent efficiency compared to their real counterparts in speech enhancement,image and signal processing.Researchers throughout the years have made many efforts to improve the learning algorithms and activation functions of CVNNs.Since CVNNs have proven to have better performance in handling the naturally complex-valued data and signals,this area of study will grow and expect the arrival of some effective improvements in the future.Therefore,there exists an obvious reason to provide a comprehensive survey paper that systematically collects and categorizes the advancement of CVNNs.In this paper,we discuss and summarize the recent advances based on their learning algorithms,activation functions,which is the most challenging part of building a CVNN,and applications.Besides,we outline the structure and applications of complex-valued convolutional,residual and recurrent neural networks.Finally,we also present some challenges and future research directions to facilitate the exploration of the ability of CVNNs.

Hyperparameter Tuning for Deep Neural Networks Based Optimization Algorithm

For training the present Neural Network(NN)models,the standard technique is to utilize decaying Learning Rates(LR).While the majority of these techniques commence with a large LR,they will decay multiple times over time.Decaying has been proved to enhance generalization as well as optimization.Other parameters,such as the network’s size,the number of hidden layers,drop-outs to avoid overfitting,batch size,and so on,are solely based on heuristics.This work has proposed Adaptive Teaching Learning Based(ATLB)Heuristic to identify the optimal hyperparameters for diverse networks.Here we consider three architec-tures Recurrent Neural Networks(RNN),Long Short Term Memory(LSTM),Bidirectional Long Short Term Memory(BiLSTM)of Deep Neural Networks for classification.The evaluation of the proposed ATLB is done through the various learning rate schedulers Cyclical Learning Rate(CLR),Hyperbolic Tangent Decay(HTD),and Toggle between Hyperbolic Tangent Decay and Triangular mode with Restarts(T-HTR)techniques.Experimental results have shown the performance improvement on the 20Newsgroup,Reuters Newswire and IMDB dataset.

Adaptive Momentum-Backpropagation Algorithm for Flood Prediction and Management in the Internet of Things

Flooding is a hazardous natural calamity that causes significant damage to lives and infrastructure in the real world.Therefore,timely and accurate decision-making is essential for mitigating flood-related damages.The traditional flood prediction techniques often encounter challenges in accuracy,timeliness,complexity in handling dynamic flood patterns and leading to substandard flood management strategies.To address these challenges,there is a need for advanced machine learning models that can effectively analyze Internet of Things(IoT)-generated flood data and provide timely and accurate flood predictions.This paper proposes a novel approach-the Adaptive Momentum and Backpropagation(AM-BP)algorithm-for flood prediction and management in IoT networks.The AM-BP model combines the advantages of an adaptive momentum technique with the backpropagation algorithm to enhance flood prediction accuracy and efficiency.Real-world flood data is used for validation,demonstrating the superior performance of the AM-BP algorithm compared to traditional methods.In addition,multilayer high-end computing architecture(MLCA)is used to handle weather data such as rainfall,river water level,soil moisture,etc.The AM-BP’s real-time abilities enable proactive flood management,facilitating timely responses and effective disaster mitigation.Furthermore,the AM-BP algorithm can analyze large and complex datasets,integrating environmental and climatic factors for more accurate flood prediction.The evaluation result shows that the AM-BP algorithm outperforms traditional approaches with an accuracy rate of 96%,96.4%F1-Measure,97%Precision,and 95.9%Recall.The proposed AM-BP model presents a promising solution for flood prediction and management in IoT networks,contributing to more resilient and efficient flood control strategies,and ensuring the safety and well-being of communities at risk of flooding.

Artificial Neural Network Methods for the Solution of Second Order Boundary Value Problems

We present a method for solving partial differential equations using artificial neural networks and an adaptive collocation strategy.In this procedure,a coarse grid of training points is used at the initial training stages,while more points are added at later stages based on the value of the residual at a larger set of evaluation points.This method increases the robustness of the neural network approximation and can result in significant computational savings,particularly when the solution is non-smooth.Numerical results are presented for benchmark problems for scalar-valued PDEs,namely Poisson and Helmholtz equations,as well as for an inverse acoustics problem.

Neural Network-Based Limiter with Transfer Learning

Recent works have shown that neural networks are promising parameter-free limiters for a variety of numerical schemes(Morgan et al.in A machine learning approach for detect-ing shocks with high-order hydrodynamic methods.et al.in J Comput Phys 367:166-191.,2018;Veiga et al.in European Conference on Computational Mechanics andⅦEuropean Conference on Computational Fluid Dynamics,vol.1,pp.2525-2550.ECCM.,2018).Following this trend,we train a neural network to serve as a shock-indicator function using simulation data from a Runge-Kutta discontinuous Galer-kin(RKDG)method and a modal high-order limiter(Krivodonova in J Comput Phys 226:879-896.,2007).With this methodology,we obtain one-and two-dimensional black-box shock-indicators which are then coupled to a standard limiter.Furthermore,we describe a strategy to transfer the shock-indicator to a residual distribution(RD)scheme without the need for a full training cycle and large data-set,by finding a mapping between the solution feature spaces from an RD scheme to an RKDG scheme,both in one-and two-dimensional problems,and on Cartesian and unstruc-tured meshes.We report on the quality of the numerical solutions when using the neural network shock-indicator coupled to a limiter,comparing its performance to traditional lim-iters,for both RKDG and RD schemes.

Neural networks-based iterative learning control consensus for periodically time-varying multi-agent systems

In this paper,the problem of adaptive iterative learning based consensus control for periodically time-varying multi-agent systems is studied,in which the dynamics of each follower are driven by nonlinearly parameterized terms with periodic disturbances.Neural networks and Fourier base expansions are introduced to describe the periodically time-varying dynamic terms.On this basis,an adaptive learning parameter with a positively convergent series term is constructed,and a distributed control protocol based on local signals between agents is designed to ensure accurate consensus of the closed-loop systems.Furthermore,consensus algorithm is generalized to solve the formation control problem.Finally,simulation experiments are implemented through MATLAB to demonstrate the effectiveness of the method used.

Corporate Credit Ratings Based on Hierarchical Heterogeneous Graph Neural Networks

order to help investors understand the credit status of target corporations and reduce investment risks,the corporate credit rating model has become an important evaluation tool in the financial market.These models are based on statistical learning,machine learning and deep learning especially graph neural networks(GNNs).However,we found that only few models take the hierarchy,heterogeneity or unlabeled data into account in the actual corporate credit rating process.Therefore,we propose a novel framework named hierarchical heterogeneous graph neural networks(HHGNN),which can fully model the hierarchy of corporate features and the heterogeneity of relationships between corporations.In addition,we design an adversarial learning block to make full use of the rich unlabeled samples in the financial data.Extensive experiments conducted on the public-listed corporate rating dataset prove that HHGNN achieves SOTA compared to the baseline methods.

Neural Network State Learning Based Adaptive Terminal ILC for Tracking Iteration-varying Target Points

Terminal iterative learning control(TILC) is developed to reduce the error between system output and a fixed desired point at the terminal end of operation interval over iterations under strictly identical initial conditions. In this work, the initial states are not required to be identical further but can be varying from iteration to iteration. In addition, the desired terminal point is not fixed any more but is allowed to change run-to-run. Consequently, a new adaptive TILC is proposed with a neural network initial state learning mechanism to achieve the learning objective over iterations. The neural network is used to approximate the effect of iteration-varying initial states on the terminal output and the neural network weights are identified iteratively along the iteration axis.A dead-zone scheme is developed such that both learning and adaptation are performed only if the terminal tracking error is outside a designated error bound. It is shown that the proposed approach is able to track run-varying terminal desired points fast with a specified tracking accuracy beyond the initial state variance.