Complex-Valued Neural Networks:A Comprehensive Survey

Complex-valued neural networks(CVNNs)have shown their excellent efficiency compared to their real counterparts in speech enhancement,image and signal processing.Researchers throughout the years have made many efforts to improve the learning algorithms and activation functions of CVNNs.Since CVNNs have proven to have better performance in handling the naturally complex-valued data and signals,this area of study will grow and expect the arrival of some effective improvements in the future.Therefore,there exists an obvious reason to provide a comprehensive survey paper that systematically collects and categorizes the advancement of CVNNs.In this paper,we discuss and summarize the recent advances based on their learning algorithms,activation functions,which is the most challenging part of building a CVNN,and applications.Besides,we outline the structure and applications of complex-valued convolutional,residual and recurrent neural networks.Finally,we also present some challenges and future research directions to facilitate the exploration of the ability of CVNNs.

Synthesization of high-capacity auto-associative memories using complex-valued neural networks

In this paper, a novel design procedure is proposed for synthesizing high-capacity auto-associative memories based on complex-valued neural networks with real-imaginary-type activation functions and constant delays. Stability criteria dependent on external inputs of neural networks are derived. The designed networks can retrieve the stored patterns by external inputs rather than initial conditions. The derivation can memorize the desired patterns with lower-dimensional neural networks than real-valued neural networks, and eliminate spurious equilibria of complex-valued neural networks. One numerical example is provided to show the effectiveness and superiority of the presented results.

GLOBAL DYNAMICS OF DELAYED BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM) NEURAL NETWORKS

Without assuming the smoothness,monotonicity and boundedness of the activation functions, some novel criteria on the existence and global exponential stability of equilibrium point for delayed bidirectional associative memory (BAM) neural networks are established by applying the Liapunov functional methods and matrix_algebraic techniques. It is shown that the new conditions presented in terms of a nonsingular M matrix described by the networks parameters,the connection matrix and the Lipschitz constant of the activation functions,are not only simple and practical,but also easier to check and less conservative than those imposed by similar results in recent literature.

Adaptive synchronization of a class of fractional-order complex-valued chaotic neural network with time-delay

This paper is concerned with the adaptive synchronization of fractional-order complex-valued chaotic neural networks(FOCVCNNs)with time-delay.The chaotic behaviors of a class of fractional-order complex-valued neural network are investigated.Meanwhile,based on the complex-valued inequalities of fractional-order derivatives and the stability theory of fractional-order complex-valued systems,a new adaptive controller and new complex-valued update laws are proposed to construct a synchronization control model for fractional-order complex-valued chaotic neural networks.Finally,the numerical simulation results are presented to illustrate the effectiveness of the developed synchronization scheme.

Multistability of delayed complex-valued recurrent neural networks with discontinuous real-imaginarytype activation functions

In this paper, the multistability issue is discussed for delayed complex-valued recurrent neural networks with discontinuous real-imaginary-type activation functions. Based on a fixed theorem and stability definition, sufficient criteria are established for the existence and stability of multiple equilibria of complex-valued recurrent neural networks. The number of stable equilibria is larger than that of real-valued recurrent neural networks, which can be used to achieve high-capacity associative memories. One numerical example is provided to show the effectiveness and superiority of the presented results.

Finite-time Mittag-Leffler synchronization of fractional-order complex-valued memristive neural networks with time delay

Without dividing the complex-valued systems into two real-valued ones, a class of fractional-order complex-valued memristive neural networks(FCVMNNs) with time delay is investigated. Firstly, based on the complex-valued sign function, a novel complex-valued feedback controller is devised to research such systems. Under the framework of Filippov solution, differential inclusion theory and Lyapunov stability theorem, the finite-time Mittag-Leffler synchronization(FTMLS) of FCVMNNs with time delay can be realized. Meanwhile, the upper bound of the synchronization settling time(SST) is less conservative than previous results. In addition, by adjusting controller parameters, the global asymptotic synchronization of FCVMNNs with time delay can also be realized, which improves and enrich some existing results. Lastly,some simulation examples are designed to verify the validity of conclusions.

Learning Dynamics of the Complex-Valued Neural Network in the Neighborhood of Singular Points

In this paper, the singularity and its effect on learning dynamics in the complex-valued neural network are elucidated. It has learned that the linear combination structure in the updating rule of the complex-valued neural network increases the speed of moving away from the singular points, and the complex-valued neural network cannot be easily influenced by the singular points, whereas the learning of the usual real-valued neural network can be attracted in the neighborhood of singular points, which causes a standstill in learning. Simulation results on the learning dynamics of the three-layered real-valued and complex-valued neural networks in the neighborhood of singularities support the analytical results.

Landslide displacement prediction based on optimized empirical mode decomposition and deep bidirectional long short-term memory network

There are two technical challenges in predicting slope deformation.The first one is the random displacement,which could not be decomposed and predicted by numerically resolving the observed accumulated displacement and time series of a landslide.The second one is the dynamic evolution of a landslide,which could not be feasibly simulated simply by traditional prediction models.In this paper,a dynamic model of displacement prediction is introduced for composite landslides based on a combination of empirical mode decomposition with soft screening stop criteria(SSSC-EMD)and deep bidirectional long short-term memory(DBi-LSTM)neural network.In the proposed model,the time series analysis and SSSC-EMD are used to decompose the observed accumulated displacements of a slope into three components,viz.trend displacement,periodic displacement,and random displacement.Then,by analyzing the evolution pattern of a landslide and its key factors triggering landslides,appropriate influencing factors are selected for each displacement component,and DBi-LSTM neural network to carry out multi-datadriven dynamic prediction for each displacement component.An accumulated displacement prediction has been obtained by a summation of each component.For accuracy verification and engineering practicability of the model,field observations from two known landslides in China,the Xintan landslide and the Bazimen landslide were collected for comparison and evaluation.The case study verified that the model proposed in this paper can better characterize the"stepwise"deformation characteristics of a slope.As compared with long short-term memory(LSTM)neural network,support vector machine(SVM),and autoregressive integrated moving average(ARIMA)model,DBi-LSTM neural network has higher accuracy in predicting the periodic displacement of slope deformation,with the mean absolute percentage error reduced by 3.063%,14.913%,and 13.960%respectively,and the root mean square error reduced by 1.951 mm,8.954 mm and 7.790 mm respectively.Conclusively,this model not only has high prediction accuracy but also is more stable,which can provide new insight for practical landslide prevention and control engineering.

Multi-Feature Fusion-Guided Multiscale Bidirectional Attention Networks for Logistics Pallet Segmentation

In the smart logistics industry,unmanned forklifts that intelligently identify logistics pallets can improve work efficiency in warehousing and transportation and are better than traditional manual forklifts driven by humans.Therefore,they play a critical role in smart warehousing,and semantics segmentation is an effective method to realize the intelligent identification of logistics pallets.However,most current recognition algorithms are ineffective due to the diverse types of pallets,their complex shapes,frequent blockades in production environments,and changing lighting conditions.This paper proposes a novel multi-feature fusion-guided multiscale bidirectional attention(MFMBA)neural network for logistics pallet segmentation.To better predict the foreground category(the pallet)and the background category(the cargo)of a pallet image,our approach extracts three types of features(grayscale,texture,and Hue,Saturation,Value features)and fuses them.The multiscale architecture deals with the problem that the size and shape of the pallet may appear different in the image in the actual,complex environment,which usually makes feature extraction difficult.Our study proposes a multiscale architecture that can extract additional semantic features.Also,since a traditional attention mechanism only assigns attention rights from a single direction,we designed a bidirectional attention mechanism that assigns cross-attention weights to each feature from two directions,horizontally and vertically,significantly improving segmentation.Finally,comparative experimental results show that the precision of the proposed algorithm is 0.53%–8.77%better than that of other methods we compared.

Seismic-inversion method for nonlinear mapping multilevel well–seismic matching based on bidirectional long short-term memory networks

In this paper,the recurrent neural network structure of a bidirectional long shortterm memory network(Bi-LSTM)with special memory cells that store information is used to characterize the deep features of the variation pattern between logging and seismic data.A mapping relationship model between high-frequency logging data and low-frequency seismic data is established via nonlinear mapping.The seismic waveform is infinitely approximated using the logging curve in the low-frequency band to obtain a nonlinear mapping model of this scale,which then stepwise approach the logging curve in the high-frequency band.Finally,a seismic-inversion method of nonlinear mapping multilevel well–seismic matching based on the Bi-LSTM network is developed.The characteristic of this method is that by applying the multilevel well–seismic matching process,the seismic data are stepwise matched to the scale range that is consistent with the logging curve.Further,the matching operator at each level can be stably obtained to effectively overcome the problems that occur in the well–seismic matching process,such as the inconsistency in the scale of two types of data,accuracy in extracting the seismic wavelet of the well-side seismic traces,and multiplicity of solutions.Model test and practical application demonstrate that this method improves the vertical resolution of inversion results,and at the same time,the boundary and the lateral characteristics of the sand body are well maintained to improve the accuracy of thin-layer sand body prediction and achieve an improved practical application effect.

Periodic Solution for a Complex-Valued Network Model with Discrete Delay

For a tridiagonal two-layer real six-neuron model,the Hopf bifurcation was investigated by studying the eigenvalue equations of the related linear system in the literature.In the present paper,we extend this two-layer real six-neuron network model into a complex-valued delayed network model.Based on the mathematical analysis method,some sufficient conditions to guarantee the existence of periodic oscillatory solutions are established under the assumption that the activation function can be separated into its real and imaginary parts.Our sufficient conditions obtained by the mathe­matical analysis method in this paper are simpler than those obtained by the Hopf bifurcation method.Computer simulation is provided to illustrate the correctness of the theoretical results.

BIDIRECTIONAL ASSOCIATIVE MEMORY ENSEMBLE

The multiple classifier system （MCS）, composed of multiple diverse classifiers or feed-forward neural networks, can significantly improve the classification or generalization ability of a single classifier. Enlightened by the fundamental idea of MCS, the ensemble is introduced into the quick learning for bidirectional associative memory （QLBAM） to construct a BAM ensemble, for improving the storage capacity and the error-correction capability without destroying the simple structure of the component BAM. Simulations show that, with an appropriate ＂overproduce and choose＂ strategy or ＂thinning＂ algorithm, the proposed BAM ensemble significantly outperforms the single QLBAM in both storage capacity and noise-tolerance capability.

Spectral transfer-learning-based metasurface design assisted by complex-valued deep neural network

Recently,deep learning has been used to establish the nonlinear and nonintuitive mapping between physical structures and electromagnetic responses of meta-atoms for higher computational efficiency.However,to obtain sufficiently accurate predictions,the conventional deep-learning-based method consumes excessive time to collect the data set,thus hindering its wide application in this interdisciplinary field.We introduce a spectral transfer-learning-based metasurface design method to achieve excellent performance on a small data set with only 1000 samples in the target waveband by utilizing open-source data from another spectral range.We demonstrate three transfer strategies and experimentally quantify their performance,among which the“frozen-none”robustly improves the prediction accuracy by∼26%compared to direct learning.We propose to use a complex-valued deep neural network during the training process to further improve the spectral predicting precision by∼30%compared to its real-valued counterparts.We design several typical teraherz metadevices by employing a hybrid inverse model consolidating this trained target network and a global optimization algorithm.The simulated results successfully validate the capability of our approach.Our work provides a universal methodology for efficient and accurate metasurface design in arbitrary wavebands,which will pave the way toward the automated and mass production of metasurfaces.

采用CNN和Bidirectional GRU的时间序列分类研究

时间序列数据具有非离散性、数据之间的时序相关性、特征空间维度大等特点,当前大多数分类方法需要经过复杂的数据处理或特征工程,未考虑到时间序列具有不同时间尺度特征以及序列数据之间的时序依赖。通过结合卷积神经网络和循环神经网络中的双向门控循环单元,提出了一个新的端对端深度学习神经网络模型BiGRU-FCN,不需要对数据进行复杂的预处理,并且通过不同的网络运算来获取多种特征信息,如卷积神经网络在时序信息上的空间特征以及双向循环神经网络在序列上的双向时序依赖特征,对单维时间序列进行分类。在大量的基准数据集上对模型进行实验与评估,实验结果表明,与现有的多种方法相比,所提出的模型具有更高的准确率,具有很好的分类效果。

Applying Stack Bidirectional LSTM Model to Intrusion Detection

Nowadays,Internet has become an indispensable part of daily life and is used in many fields.Due to the large amount of Internet traffic,computers are subject to various security threats,which may cause serious economic losses and even endanger national security.It is hoped that an effective security method can systematically classify intrusion data in order to avoid leakage of important data or misuse of data.As machine learning technology matures,deep learning is widely used in various industries.Combining deep learning with network security and intrusion detection is the current trend.In this paper,the problem of data classification in intrusion detection system is studied.We propose an intrusion detection model based on stack bidirectional long short-term memory(LSTM),introduce stack bidirectional LSTM into the field of intrusion detection and apply it to the intrusion detection.In order to determine the appropriate parameters and structure of stack bidirectional LSTM network,we have carried out experiments on various network structures and parameters and analyzed the experimental results.The classic KDD Cup’1999 dataset was selected for experiments so that we can obtain convincing and comparable results.Experimental results derived from the KDD Cup’1999 dataset show that the network with three hidden layers containing 80 LSTM cells is superior to other algorithms in computational cost and detection performance due to stack bidirectional LSTM model’s ability to review time and correlate with connected records continuously.The experiment shows the effectiveness of stack bidirectional LSTM network in intrusion detection.

Physics-informed neural network approach for heat generation rate estimation of lithium-ion battery under various driving conditions

Accurate insight into the heat generation rate(HGR) of lithium-ion batteries(LIBs) is one of key issues for battery management systems to formulate thermal safety warning strategies in advance.For this reason,this paper proposes a novel physics-informed neural network(PINN) approach for HGR estimation of LIBs under various driving conditions.Specifically,a single particle model with thermodynamics(SPMT) is first constructed for extracting the critical physical knowledge related with battery HGR.Subsequently,the surface concentrations of positive and negative electrodes in battery SPMT model are integrated into the bidirectional long short-term memory(BiLSTM) networks as physical information.And combined with other feature variables,a novel PINN approach to achieve HGR estimation of LIBs with higher accuracy is constituted.Additionally,some critical hyperparameters of BiLSTM used in PINN approach are determined through Bayesian optimization algorithm(BOA) and the results of BOA-based BiLSTM are compared with other traditional BiLSTM/LSTM networks.Eventually,combined with the HGR data generated from the validated virtual battery,it is proved that the proposed approach can well predict the battery HGR under the dynamic stress test(DST) and worldwide light vehicles test procedure(WLTP),the mean absolute error under DST is 0.542 kW/m^(3),and the root mean square error under WLTP is1.428 kW/m^(3)at 25℃.Lastly,the investigation results of this paper also show a new perspective in the application of the PINN approach in battery HGR estimation.

LMI-based approach for global asymptotic stability analysis of continuous BAM neural networks

Studies on the stability of the equilibrium points of continuous bidirectional associative memory (BAM) neural network have yielded many useful results. A novel neural network model called standard neural network model (SNNM) is ad- vanced. By using state affine transformation, the BAM neural networks were converted to SNNMs. Some sufficient conditions for the global asymptotic stability of continuous BAM neural networks were derived from studies on the SNNMs’ stability. These conditions were formulated as easily verifiable linear matrix inequalities (LMIs), whose conservativeness is relatively low. The approach proposed extends the known stability results, and can also be applied to other forms of recurrent neural networks (RNNs).

Stability analysis of discrete-time BAM neural networks based on standard neural network models

To facilitate stability analysis of discrete-time bidirectional associative memory (BAM) neural networks, they were converted into novel neural network models, termed standard neural network models (SNNMs), which interconnect linear dynamic systems and bounded static nonlinear operators. By combining a number of different Lyapunov functionals with S-procedure, some useful criteria of global asymptotic stability and global exponential stability of the equilibrium points of SNNMs were derived. These stability conditions were formulated as linear matrix inequalities (LMIs). So global stability of the discrete-time BAM neural networks could be analyzed by using the stability results of the SNNMs. Compared to the existing stability analysis methods, the proposed approach is easy to implement, less conservative, and is applicable to other recurrent neural networks.

Robust asymptotic stability for BAM neural networks with time-varying delays via LMI approach

Several novel stability conditions for BAM neural networks with time-varying delays are studied.Based on Lyapunov-Krasovskii functional combined with linear matrix inequality approach,the delay-dependent linear matrix inequality（LMI） conditions are established to guarantee robust asymptotic stability for given delayed BAM neural networks.These criteria can be easily verified by utilizing the recently developed algorithms for solving LMIs.A numerical example is provided to demonstrate the effectiveness and less conservatism of the main results.