Improving Performance of Recurrent Neural Networks Using Simulated Annealing for Vertical Wind Speed Estimation

An accurate vertical wind speed(WS)data estimation is required to determine the potential for wind farm installation.In general,the vertical extrapolation of WS at different heights must consider different parameters fromdifferent locations,such as wind shear coefficient,roughness length,and atmospheric conditions.The novelty presented in this article is the introduction of two steps optimization for the Recurrent Neural Networks(RNN)model to estimate WS at different heights using measurements from lower heights.The first optimization of the RNN is performed to minimize a differentiable cost function,namely,mean squared error(MSE),using the Broyden-Fletcher-Goldfarb-Shanno algorithm.Secondly,the RNN is optimized to reduce a non-differentiable cost function using simulated annealing(RNN-SA),namely mean absolute error(MAE).Estimation ofWS vertically at 50 m height is done by training RNN-SA with the actualWS data a 10–40 m heights.The estimatedWS at height of 50 m and the measured WS at 10–40 heights are further used to train RNN-SA to obtain WS at 60 m height.This procedure is repeated continuously until theWS is estimated at a height of 180 m.The RNN-SA performance is compared with the standard RNN,Multilayer Perceptron(MLP),Support Vector Machine(SVM),and state of the art methods like convolutional neural networks(CNN)and long short-term memory(LSTM)networks to extrapolate theWS vertically.The estimated values are also compared with realWS dataset acquired using LiDAR and tested using four error metrics namely,mean squared error(MSE),mean absolute percentage error(MAPE),mean bias error(MBE),and coefficient of determination(R2).The numerical experimental results show that the MSE values between the estimated and actualWS at 180mheight for the RNN-SA,RNN,MLP,and SVM methods are found to be 2.09,2.12,2.37,and 2.63,respectively.

Machine learning for pore-water pressure time-series prediction:Application of recurrent neural networks

Knowledge of pore-water pressure(PWP)variation is fundamental for slope stability.A precise prediction of PWP is difficult due to complex physical mechanisms and in situ natural variability.To explore the applicability and advantages of recurrent neural networks(RNNs)on PWP prediction,three variants of RNNs,i.e.,standard RNN,long short-term memory(LSTM)and gated recurrent unit(GRU)are adopted and compared with a traditional static artificial neural network(ANN),i.e.,multi-layer perceptron(MLP).Measurements of rainfall and PWP of representative piezometers from a fully instrumented natural slope in Hong Kong are used to establish the prediction models.The coefficient of determination(R^2)and root mean square error(RMSE)are used for model evaluations.The influence of input time series length on the model performance is investigated.The results reveal that MLP can provide acceptable performance but is not robust.The uncertainty bounds of RMSE of the MLP model range from 0.24 kPa to 1.12 k Pa for the selected two piezometers.The standard RNN can perform better but the robustness is slightly affected when there are significant time lags between PWP changes and rainfall.The GRU and LSTM models can provide more precise and robust predictions than the standard RNN.The effects of the hidden layer structure and the dropout technique are investigated.The single-layer GRU is accurate enough for PWP prediction,whereas a double-layer GRU brings extra time cost with little accuracy improvement.The dropout technique is essential to overfitting prevention and improvement of accuracy.

OTT Messages Modeling and Classification Based on Recurrent Neural Networks

A vast amount of information has been produced in recent years,which brings a huge challenge to information management.The better usage of big data is of important theoretical and practical significance for effectively addressing and managing messages.In this paper,we propose a nine-rectangle-grid information model according to the information value and privacy,and then present information use policies based on the rough set theory.Recurrent neural networks were employed to classify OTT messages.The content of user interest is effectively incorporated into the classification process during the annotation of OTT messages,ending with a reliable trained classification model.Experimental results showed that the proposed method yielded an accurate classification performance and hence can be used for effective distribution and control of OTT messages.

New Stability Criteria for Recurrent Neural Networks with a Time-varying Delay

This paper deals with the stability of static recurrent neural networks （RNNs） with a time-varying delay. An augmented Lyapunov-Krasovskii functional is employed, in which some useful terms are included. Furthermore, the relationship among the timevarying delay, its upper bound and their difierence, is taken into account, and novel bounding techniques for 1- τ（t） are employed. As a result, without ignoring any useful term in the derivative of the Lyapunov-Krasovskii functional, the resulting delay-dependent criteria show less conservative than the existing ones. Finally, a numerical example is given to demonstrate the effectiveness of the proposed methods.

Non-Minimum Phase Nonlinear System Predictive Control Based on Local Recurrent Neural Networks

After a recursive multi-step-ahead predictor for nonlinear systems based on local recurrent neural networks is introduced, an intelligent FID controller is adopted to correct the errors including identified model errors and accumulated errors produced in the recursive process. Characterized by predictive control, this method can achieve a good control accuracy and has good robustness. A simulation study shows that this control algorithm is very effective.

Robust exponential stability analysis of a larger class of discrete-time recurrent neural networks

The robust exponential stability of a larger class of discrete-time recurrent neural networks (RNNs) is explored in this paper. A novel neural network model, named standard neural network model (SNNM), is introduced to provide a general framework for stability analysis of RNNs. Most of the existing RNNs can be transformed into SNNMs to be analyzed in a unified way. Applying Lyapunov stability theory method and S-Procedure technique, two useful criteria of robust exponential stability for the discrete-time SNNMs are derived. The conditions presented are formulated as linear matrix inequalities (LMIs) to be easily solved using existing efficient convex optimization techniques. An example is presented to demonstrate the transformation procedure and the effectiveness of the results.

Robust stability analysis of Takagi-Sugeno uncertain stochastic fuzzy recurrent neural networks with mixed time-varying delays

In this paper, the global stability of Takagi-Sugeno （TS） uncertain stochastic fuzzy recurrent neural networks with discrete and distributed time-varying delays （TSUSFRNNs） is considered. A novel LMI-based stability criterion is obtained by using Lyapunov functional theory to guarantee the asymptotic stability of TSUSFRNNs. The proposed stability conditions are demonstrated through numerical examples. Furthermore, the supplementary requirement that the time derivative of time-varying delays must be smaller than one is removed. Comparison results are demonstrated to show that the proposed method is more able to guarantee the widest stability region than the other methods available in the existing literature.

Visual analytics tool for the interpretation of hidden states in recurrent neural networks

In this paper,we introduce a visual analytics approach aimed at helping machine learning experts analyze the hidden states of layers in recurrent neural networks.Our technique allows the user to interactively inspect how hidden states store and process information throughout the feeding of an input sequence into the network.The technique can help answer questions,such as which parts of the input data have a higher impact on the prediction and how the model correlates each hidden state configuration with a certain output.Our visual analytics approach comprises several components:First,our input visualization shows the input sequence and how it relates to the output(using color coding).In addition,hidden states are visualized through a nonlinear projection into a 2-D visualization space using t-distributed stochastic neighbor embedding to understand the shape of the space of the hidden states.Trajectories are also employed to show the details of the evolution of the hidden state configurations.Finally,a time-multi-class heatmap matrix visualizes the evolution of the expected predictions for multi-class classifiers,and a histogram indicates the distances between the hidden states within the original space.The different visualizations are shown simultaneously in multiple views and support brushing-and-linking to facilitate the analysis of the classifications and debugging for misclassified input sequences.To demonstrate the capability of our approach,we discuss two typical use cases for long short-term memory models applied to two widely used natural language processing datasets.

Exponential stability and existence of periodic solutions for a class of recurrent neural networks with delays

Both the global exponential stability and the existence of periodic solutions for a class of recurrent neural networks with continuously distributed delays （RNNs） are studied. By employing the inequality α∏k=1^m bk^qk≤1/r ∑qkbk^r＋1/rα^r（α≥0,bk≥0,qk〉0,with ∑k=1^m qk=r-1,r≥1, constructing suitable Lyapunov r k=l k=l functions and applying the homeomorphism theory, a family of simple and new sufficient conditions are given ensuring the global exponential stability and the existence of periodic solutions of RNNs. The results extend and improve the results of earlier publications.

STOCHASTIC STABILITY OF UNCERTAIN RECURRENT NEURAL NETWORKS WITH MARKOVIAN JUMPING PARAMETERS

In this paper, global robust stability of uncertain stochastic recurrent neural networks with Markovian jumping parameters is considered. A novel Linear matrix inequal- ity（LMI） based stability criterion is obtained to guarantee the asymptotic stability of uncertain stochastic recurrent neural networks with Markovian jumping parameters. The results are derived by using the Lyapunov functional technique, Lipchitz condition and S-procuture. Finally, numerical examples are given to demonstrate the correctness of the theoretical results. Our results are also compared with results discussed in [31] and [34] to show the effectiveness and conservativeness.

Delay dependent stability criteria for recurrent neural networks with time varying delays

This paper aims to present some delay-dependent global asymptotic stability criteria for recurrent neural networks with time varying delays. The obtained results have no restriction on the magnitude of derivative of time varying delay, and can be easily checked due to the form of linear matrix inequality. By comparison with some previous results, the obtained results are less conservative. A numerical example is utilized to demonstrate the effectiveness of the obtained results.

Adaptive learning with guaranteed stability for discrete-time recurrent neural networks

To avoid unstable learning, a stable adaptive learning algorithm was proposed for discrete-time recurrent neural networks. Unlike the dynamic gradient methods, such as the backpropagation through time and the real time recurrent learning, the weights of the recurrent neural networks were updated online in terms of Lyapunov stability theory in the proposed learning algorithm, so the learning stability was guaranteed. With the inversion of the activation function of the recurrent neural networks, the proposed learning algorithm can be easily implemented for solving varying nonlinear adaptive learning problems and fast convergence of the adaptive learning process can be achieved. Simulation experiments in pattern recognition show that only 5 iterations are needed for the storage of a 15×15 binary image pattern and only 9 iterations are needed for the perfect realization of an analog vector by an equilibrium state with the proposed learning algorithm.

Global stability of interval recurrent neural networks

The robust global exponential stability of a class of interval recurrent neural networks（RNNs） is studied,and a new robust stability criterion is obtained in the form of linear matrix inequality.The problem of robust stability of interval RNNs is transformed into a problem of solving a class of linear matrix inequalities.Thus,the robust stability of interval RNNs can be analyzed by directly using the linear matrix inequalities（LMI） toolbox of MATLAB.Numerical example is given to show the effectiveness of the obtained results.

Improved delay-dependent globally asymptotic stability of delayed uncertain recurrent neural networks with Markovian jumping parameters

In this paper, we have improved delay-dependent stability criteria for recurrent neural networks with a delay varying over a range and Markovian jumping parameters. The criteria improve over some previous ones in that they have fewer matrix variables yet less conservatism. In addition, a numerical example is provided to illustrate the applicability of the result using the linear matrix inequality toolbox in MATLAB.

ADAPTIVE RECURRENT NEURAL NETWORKS TRACKING-FILTER FOR MANEUVERING TARGET

It is a challenge to track the maneuvering targets with noise disturbance and unknown dynamics. In this paper, an adaptive recurrent neural network tracking filter (ARNNF) for use in maneuvering target tracking was provided. The scheme is based on recurrent neural networks of which the recurrence provides a potentially unlimited memory depth adjusted by the network adaptively ( i.e. , it finds the best duration to represent the input signals past), and thus can actually capture the dynamics of the system that produced a temporal signal. On the other hand, recurrent neural network can approximate arbitrary nonlinear functions in L 2 space. The theoretical analysis indicates that the ARNNF can track the maneuvering targets with optimal filtering performance. Comparisons with IMM and AIMM algorithm show that ARNNF has better performance, and furthermore the ARNNF does not rely on the assumption with the known maneuvering target models, measurement noise and system noise.

INFLUENCE OF NOISE AND DELAY ON REACTION-DIFFUSION RECURRENT NEURAL NETWORKS

In this paper, the influence of the noise and delay upon the stability property of reaction-diffusion recurrent neural networks （RNNs） with the time-varying delay is discussed. The new and easily verifiable conditions to guarantee the mean value exponential stability of an equilibrium solution are derived. The rate of exponential convergence can be estimated by means of a simple computation based on these criteria.

Novel delay-distribution-dependent stability analysis for continuous-time recurrent neural networks with stochastic delay

In this paper, the problem of delay-distribution-dependent stability is investigated for continuous-time recurrent neural networks （CRNNs） with stochastic delay. Different from the common assumptions on time delays, it is assumed that the probability distribution of the delay taking values in some intervals is known a priori. By making full use of the information concerning the probability distribution of the delay and by using a tighter bounding technique （the reciprocally convex combination method）, less conservative asymptotic mean-square stable sufficient conditions are derived in terms of linear matrix inequalities （LMIs）. Two numerical examples show that our results are better than the existing ones.

Mean square stability of recurrent neural networks with random delay and Markovian switching

To establish easily proved conditions under which the random delayed recurrent neural network with Markovian switching is mean-square stability,the evolution of the delay was modeled by a continuous-time homogeneous Markov process with a finite number of states.By employing Lyapunov-Krasovskii functionals and conducting stochastic analysis,a linear matrix inequality (LMI) approach was developed to derive the criteria for mean-square stability,which can be readily checked by some standard numerical packages such as the Matlab LMI Toolbox.A numerical example was exploited to show the usefulness of the derived LMI-based stability conditions.

Remaining Useful Life Prediction for a Roller in a Hot Strip Mill Based on Deep Recurrent Neural Networks

Accurate estimation of the remaining useful life(RUL)and health state for rollers is of great significance to hot rolling production.It can provide decision support for roller management so as to improve the productivity of the hot rolling process.In addition,the RUL prediction for rollers is helpful in transitioning from the current regular maintenance strategy to conditional-based maintenance.Therefore,a new method that can extract coarse-grained and fine-grained features from batch data to predict the RUL of the rollers is proposed in this paper.Firstly,a new deep learning network architecture based on recurrent neural networks that can make full use of the extracted coarsegrained fine-grained features to estimate the heath indicator(HI)is developed,where the HI is able to indicate the health state of the roller.Following that,a state-space model is constructed to describe the HI,and the probabilistic distribution of RUL can be estimated by extrapolating the HI degradation model to a predefined failure threshold.Finally,application to a hot strip mill is given to verify the effectiveness of the proposed methods using data collected from an industrial site,and the relatively low RMSE and MAE values demonstrate its advantages compared with some other popular deep learning methods.

Multistability of delayed complex-valued recurrent neural networks with discontinuous real-imaginarytype activation functions

In this paper, the multistability issue is discussed for delayed complex-valued recurrent neural networks with discontinuous real-imaginary-type activation functions. Based on a fixed theorem and stability definition, sufficient criteria are established for the existence and stability of multiple equilibria of complex-valued recurrent neural networks. The number of stable equilibria is larger than that of real-valued recurrent neural networks, which can be used to achieve high-capacity associative memories. One numerical example is provided to show the effectiveness and superiority of the presented results.