Sparse Autoencoder-based Multi-head Deep Neural Networks for Machinery Fault Diagnostics with Detection of Novelties

Supervised fault diagnosis typically assumes that all the types of machinery failures are known.However,in practice unknown types of defect,i.e.,novelties,may occur,whose detection is a challenging task.In this paper,a novel fault diagnostic method is developed for both diagnostics and detection of novelties.To this end,a sparse autoencoder-based multi-head Deep Neural Network(DNN)is presented to jointly learn a shared encoding representation for both unsupervised reconstruction and supervised classification of the monitoring data.The detection of novelties is based on the reconstruction error.Moreover,the computational burden is reduced by directly training the multi-head DNN with rectified linear unit activation function,instead of performing the pre-training and fine-tuning phases required for classical DNNs.The addressed method is applied to a benchmark bearing case study and to experimental data acquired from a delta 3D printer.The results show that its performance is satisfactory both in detection of novelties and fault diagnosis,outperforming other state-of-the-art methods.This research proposes a novel fault diagnostics method which can not only diagnose the known type of defect,but also detect unknown types of defects.

Physics-informed neural network-based petroleum reservoir simulation with sparse data using domain decomposition

Recent advances in deep learning have expanded new possibilities for fluid flow simulation in petroleum reservoirs.However,the predominant approach in existing research is to train neural networks using high-fidelity numerical simulation data.This presents a significant challenge because the sole source of authentic wellbore production data for training is sparse.In response to this challenge,this work introduces a novel architecture called physics-informed neural network based on domain decomposition(PINN-DD),aiming to effectively utilize the sparse production data of wells for reservoir simulation with large-scale systems.To harness the capabilities of physics-informed neural networks(PINNs)in handling small-scale spatial-temporal domain while addressing the challenges of large-scale systems with sparse labeled data,the computational domain is divided into two distinct sub-domains:the well-containing and the well-free sub-domain.Moreover,the two sub-domains and the interface are rigorously constrained by the governing equations,data matching,and boundary conditions.The accuracy of the proposed method is evaluated on two problems,and its performance is compared against state-of-the-art PINNs through numerical analysis as a benchmark.The results demonstrate the superiority of PINN-DD in handling large-scale reservoir simulation with limited data and show its potential to outperform conventional PINNs in such scenarios.

Sparse Seismic Data Reconstruction Based on a Convolutional Neural Network Algorithm

At present,the acquisition of seismic data is developing toward high-precision and high-density methods.However,complex natural environments and cultural factors in many exploration areas cause difficulties in achieving uniform and intensive acquisition,which makes complete seismic data collection impossible.Therefore,data reconstruction is required in the processing link to ensure imaging accuracy.Deep learning,as a new field in rapid development,presents clear advantages in feature extraction and modeling.In this study,the convolutional neural network deep learning algorithm is applied to seismic data reconstruction.Based on the convolutional neural network algorithm and combined with the characteristics of seismic data acquisition,two training strategies of supervised and unsupervised learning are designed to reconstruct sparse acquisition seismic records.First,a supervised learning strategy is proposed for labeled data,wherein the complete seismic data are segmented as the input of the training set and are randomly sampled before each training,thereby increasing the number of samples and the richness of features.Second,an unsupervised learning strategy based on large samples is proposed for unlabeled data,and the rolling segmentation method is used to update(pseudo)labels and training parameters in the training process.Through the reconstruction test of simulated and actual data,the deep learning algorithm based on a convolutional neural network shows better reconstruction quality and higher accuracy than compressed sensing based on Curvelet transform.

Application of graph neural network and feature information enhancement in relation inference of sparse knowledge graph

At present,knowledge embedding methods are widely used in the field of knowledge graph(KG)reasoning,and have been successfully applied to those with large entities and relationships.However,in research and production environments,there are a large number of KGs with a small number of entities and relations,which are called sparse KGs.Limited by the performance of knowledge extraction methods or some other reasons(some common-sense information does not appear in the natural corpus),the relation between entities is often incomplete.To solve this problem,a method of the graph neural network and information enhancement is proposed.The improved method increases the mean reciprocal rank(MRR)and Hit@3 by 1.6%and 1.7%,respectively,when the sparsity of the FB15K-237 dataset is 10%.When the sparsity is 50%,the evaluation indexes MRR and Hit@10 are increased by 0.8%and 1.8%,respectively.

Prediction Model of Aircraft Icing Based on Deep Neural Network

Icing is an important factor threatening aircraft flight safety.According to the requirements of airworthiness regulations,aircraft icing safety assessment is needed to be carried out based on the ice shapes formed under different icing conditions.Due to the complexity of the icing process,the rapid assessment of ice shape remains an important challenge.In this paper,an efficient prediction model of aircraft icing is established based on the deep belief network(DBN)and the stacked auto-encoder(SAE),which are all deep neural networks.The detailed network structures are designed and then the networks are trained according to the samples obtained by the icing numerical computation.After that the model is applied on the ice shape evaluation of NACA0012 airfoil.The results show that the model can accurately capture the nonlinear behavior of aircraft icing and thus make an excellent ice shape prediction.The model provides an important tool for aircraft icing analysis.

Source Recovery in Underdetermined Blind Source Separation Based on Artificial Neural Network

We propose a novel source recovery algorithm for underdetermined blind source separation, which can result in better accuracy and lower computational cost. On the basis of the model of underdetermined blind source separation, the artificial neural network with single-layer perceptron is introduced into the proposed algorithm. Source signals are regarded as the weight vector of single-layer perceptron, and approximate ι~0-norm is taken into account for output error decision rule of the perceptron, which leads to the sparse recovery. Then the procedure of source recovery is adjusting the weight vector of the perceptron. What's more, the optimal learning factor is calculated and a descent sequence of smoothed parameter is used during iteration, which improves the performance and significantly decreases computational complexity of the proposed algorithm. The simulation results reveal that the algorithm proposed can recover the source signal with high precision, while it requires lower computational cost.

Alzheimer’s Disease Stage Classification Using a Deep Transfer Learning and Sparse Auto Encoder Method

Alzheimer’s Disease(AD)is a progressive neurological disease.Early diagnosis of this illness using conventional methods is very challenging.Deep Learning(DL)is one of the finest solutions for improving diagnostic procedures’performance and forecast accuracy.The disease’s widespread distribution and elevated mortality rate demonstrate its significance in the older-onset and younger-onset age groups.In light of research investigations,it is vital to consider age as one of the key criteria when choosing the subjects.The younger subjects are more susceptible to the perishable side than the older onset.The proposed investigation concentrated on the younger onset.The research used deep learning models and neuroimages to diagnose and categorize the disease at its early stages automatically.The proposed work is executed in three steps.The 3D input images must first undergo image pre-processing using Weiner filtering and Contrast Limited Adaptive Histogram Equalization(CLAHE)methods.The Transfer Learning(TL)models extract features,which are subsequently compressed using cascaded Auto Encoders(AE).The final phase entails using a Deep Neural Network(DNN)to classify the phases of AD.The model was trained and tested to classify the five stages of AD.The ensemble ResNet-18 and sparse autoencoder with DNN model achieved an accuracy of 98.54%.The method is compared to state-of-the-art approaches to validate its efficacy and performance.

Learning sparse and smooth functions by deep Sigmoid nets

To pursue the outperformance of deep nets in learning,we construct a deep net with three hidden layers and prove that,implementing the empirical risk minimization(ERM)on this deep net,the estimator can theoretically realize the optimal learning rates without the classical saturation problem.In other words,deepening the networks with only three hidden layers can overcome the saturation and not degrade the optimal learning rates.The obtained results underlie the success of deep nets and provide a theoretical guidance for deep learning.

Multi-Layer Deep Sparse Representation for Biological Slice Image Inpainting

Biological slices are an effective tool for studying the physiological structure and evolutionmechanism of biological systems.However,due to the complexity of preparation technology and the presence of many uncontrollable factors during the preparation processing,leads to problems such as difficulty in preparing slice images and breakage of slice images.Therefore,we proposed a biological slice image small-scale corruption inpainting algorithm with interpretability based on multi-layer deep sparse representation,achieving the high-fidelity reconstruction of slice images.We further discussed the relationship between deep convolutional neural networks and sparse representation,ensuring the high-fidelity characteristic of the algorithm first.A novel deep wavelet dictionary is proposed that can better obtain image prior and possess learnable feature.And multi-layer deep sparse representation is used to implement dictionary learning,acquiring better signal expression.Compared with methods such as NLABH,Shearlet,Partial Differential Equation(PDE),K-Singular Value Decomposition(K-SVD),Convolutional Sparse Coding,and Deep Image Prior,the proposed algorithm has better subjective reconstruction and objective evaluation with small-scale image data,which realized high-fidelity inpainting,under the condition of small-scale image data.And theOn2-level time complexitymakes the proposed algorithm practical.The proposed algorithm can be effectively extended to other cross-sectional image inpainting problems,such as magnetic resonance images,and computed tomography images.

Deep Neural Network Based Detection and Segmentation of Ships for Maritime Surveillance

In recent years,computer visionfinds wide applications in maritime surveillance with its sophisticated algorithms and advanced architecture.Auto-matic ship detection with computer vision techniques provide an efficient means to monitor as well as track ships in water bodies.Waterways being an important medium of transport require continuous monitoring for protection of national security.The remote sensing satellite images of ships in harbours and water bodies are the image data that aid the neural network models to localize ships and to facilitate early identification of possible threats at sea.This paper proposes a deep learning based model capable enough to classify between ships and no-ships as well as to localize ships in the original images using bounding box tech-nique.Furthermore,classified ships are again segmented with deep learning based auto-encoder model.The proposed model,in terms of classification,provides suc-cessful results generating 99.5%and 99.2%validation and training accuracy respectively.The auto-encoder model also produces 85.1%and 84.2%validation and training accuracies.Moreover the IoU metric of the segmented images is found to be of 0.77 value.The experimental results reveal that the model is accu-rate and can be implemented for automatic ship detection in water bodies consid-ering remote sensing satellite images as input to the computer vision system.

Anomaly-Based Intrusion DetectionModel Using Deep Learning for IoT Networks

The rapid growth of Internet of Things(IoT)devices has brought numerous benefits to the interconnected world.However,the ubiquitous nature of IoT networks exposes them to various security threats,including anomaly intrusion attacks.In addition,IoT devices generate a high volume of unstructured data.Traditional intrusion detection systems often struggle to cope with the unique characteristics of IoT networks,such as resource constraints and heterogeneous data sources.Given the unpredictable nature of network technologies and diverse intrusion methods,conventional machine-learning approaches seem to lack efficiency.Across numerous research domains,deep learning techniques have demonstrated their capability to precisely detect anomalies.This study designs and enhances a novel anomaly-based intrusion detection system(AIDS)for IoT networks.Firstly,a Sparse Autoencoder(SAE)is applied to reduce the high dimension and get a significant data representation by calculating the reconstructed error.Secondly,the Convolutional Neural Network(CNN)technique is employed to create a binary classification approach.The proposed SAE-CNN approach is validated using the Bot-IoT dataset.The proposed models exceed the performance of the existing deep learning approach in the literature with an accuracy of 99.9%,precision of 99.9%,recall of 100%,F1 of 99.9%,False Positive Rate(FPR)of 0.0003,and True Positive Rate(TPR)of 0.9992.In addition,alternative metrics,such as training and testing durations,indicated that SAE-CNN performs better.

Intrusion Detection through DCSYS Propagation Compared to Auto-encoders

In network settings,one of the major disadvantages that threaten the network protocols is the insecurity.In most cases,unscrupulous people or bad actors can access information through unsecured connections by planting software or what we call malicious software otherwise anomalies.The presence of anomalies is also one of the disadvantages,internet users are constantly plagued by virus on their system and get activated when a harmless link is clicked on,this a case of true benign detected as false.Deep learning is very adept at dealing with such cases,but sometimes it has its own faults when dealing benign cases.Here we tend to adopt a dynamic control system(DCSYS)that addresses data packets based on benign scenario to truly report on false benign and exclude anomalies.Its performance is compared with artificial neural network auto-encoders to define its predictive power.Results show that though physical systems can adapt securely,it can be used for network data packets to identify true benign cases.

Performance Evaluation of Super-Resolution Methods Using Deep-Learning and Sparse-Coding for Improving the Image Quality of Magnified Images in Chest Radiographs

Purpose: To detect small diagnostic signals such as lung nodules in chest radiographs, radiologists magnify a region-of-interest using linear interpolation methods. However, such methods tend to generate over-smoothed images with artifacts that can make interpretation difficult. The purpose of this study was to investigate the effectiveness of super-resolution methods for improving the image quality of magnified chest radiographs. Materials and Methods: A total of 247 chest X-rays were sampled from the JSRT database, then divided into 93 training cases with non-nodules and 154 test cases with lung nodules. We first trained two types of super-resolution methods, sparse-coding super-resolution (ScSR) and super-resolution convolutional neural network (SRCNN). With the trained super-resolution methods, the high-resolution image was then reconstructed using the super-resolution methods from a low-resolution image that was down-sampled from the original test image. We compared the image quality of the super-resolution methods and the linear interpolations (nearest neighbor and bilinear interpolations). For quantitative evaluation, we measured two image quality metrics: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). For comparative evaluation of the super-resolution methods, we measured the computation time per image. Results: The PSNRs and SSIMs for the ScSR and the SRCNN schemes were significantly higher than those of the linear interpolation methods (p p p Conclusion: Super-resolution methods provide significantly better image quality than linear interpolation methods for magnified chest radiograph images. Of the two tested schemes, the SRCNN scheme processed the images fastest;thus, SRCNN could be clinically superior for processing radiographs in terms of both image quality and processing speed.

Sparsity-Aware Channel Estimation for mmWave Massive MIMO: A Deep CNN-Based Approach

The deep convolutional neural network(CNN)is exploited in this work to conduct the challenging channel estimation for mmWave massive multiple input multiple output(MIMO)systems.The inherent sparse features of the mmWave massive MIMO channels can be extracted and the sparse channel supports can be learnt by the multi-layer CNN-based network through training.Then accurate channel inference can be efficiently implemented using the trained network.The estimation accuracy and spectrum efficiency can be further improved by fully utilizing the spatial correlation among the sparse channel supports of different antennas.It is verified by simulation results that the proposed deep CNN-based scheme significantly outperforms the state-of-the-art benchmarks in both accuracy and spectrum efficiency.

Deep Learning-Based Intrusion System for Vehicular Ad Hoc Networks

The increasing use of the Internet with vehicles has made travel more convenient.However,hackers can attack intelligent vehicles through various technical loopholes,resulting in a range of security issues.Due to these security issues,the safety protection technology of the in-vehicle system has become a focus of research.Using the advanced autoencoder network and recurrent neural network in deep learning,we investigated the intrusion detection system based on the in-vehicle system.We combined two algorithms to realize the efficient learning of the vehicle’s boundary behavior and the detection of intrusive behavior.In order to verify the accuracy and efficiency of the proposed model,it was evaluated using real vehicle data.The experimental results show that the combination of the two technologies can effectively and accurately identify abnormal boundary behavior.The parameters of the model are self-iteratively updated using the time-based back propagation algorithm.We verified that the model proposed in this study can reach a nearly 96%accurate detection rate.

A comparative evaluation of Stacked Auto-Encoder neural network and Multi-Layer Extreme Learning Machine for detection and classification of faults in transmission lines using WAMS data

Smart grid is envisaged as a power grid that is extremely reliable and flexible.The electrical grid has wide-area measuring devices like Phasor measurement units(PMUs)deployed to provide real-time grid information and resolve issues effectively and speedily without compromising system availability.The development and application of machine learning approaches for power system protection and state estimation have been facilitated by the availability of measurement data.This research proposes a transmission line fault detection and classification(FD&C)system based on an auto-encoder neural network.A comparison between a Multi-Layer Extreme Learning Machine(ML-ELM)network model and a Stacked Auto-Encoder neural network(SAE)is made.Additionally,the performance of the models developed is compared to that of state-of-the-art classifier models employing feature datasets acquired by wavelet transform based feature extraction as well as other deep learning models.With substantially shorter testing time,the suggested auto-encoder models detect faults with 100% accuracy and classify faults with 99.92% and 99.79%accuracy.The computational efficiency of the ML-ELM model is demonstrated with high accuracy of classification with training time and testing time less than 50 ms.To emulate real system scenarios the models are developed with datasets with noise with signal-to-noise-ratio(SNR)ranging from 10 dB to 40 dB.The efficacy of the models is demonstrated with data from the IEEE 39 bus test system.

Attention-Guided Sparse Adversarial Attacks with Gradient Dropout

Deep neural networks are extremely vulnerable to externalities from intentionally generated adversarial examples which are achieved by overlaying tiny noise on the clean images.However,most existing transfer-based attack methods are chosen to add perturbations on each pixel of the original image with the same weight,resulting in redundant noise in the adversarial examples,which makes them easier to be detected.Given this deliberation,a novel attentionguided sparse adversarial attack strategy with gradient dropout that can be readily incorporated with existing gradient-based methods is introduced to minimize the intensity and the scale of perturbations and ensure the effectiveness of adversarial examples at the same time.Specifically,in the gradient dropout phase,some relatively unimportant gradient information is randomly discarded to limit the intensity of the perturbation.In the attentionguided phase,the influence of each pixel on the model output is evaluated by using a soft mask-refined attention mechanism,and the perturbation of those pixels with smaller influence is limited to restrict the scale of the perturbation.After conducting thorough experiments on the NeurIPS 2017 adversarial dataset and the ILSVRC 2012 validation dataset,the proposed strategy holds the potential to significantly diminish the superfluous noise present in adversarial examples,all while keeping their attack efficacy intact.For instance,in attacks on adversarially trained models,upon the integration of the strategy,the average level of noise injected into images experiences a decline of 8.32%.However,the average attack success rate decreases by only 0.34%.Furthermore,the competence is possessed to substantially elevate the attack success rate by merely introducing a slight degree of perturbation.

融合稀疏八叉树与卷积神经网络的汽车风阻系数预测

针对汽车风阻系数预测研究中参数化方法难以准确表征汽车外造型的难题,提出融合稀疏八叉树与卷积神经网络的汽车风阻系数预测方法。将汽车外造型按照八叉树结构离散,使用平均法向量对离散的复杂曲面进行简化,利用卷积神经网络对八叉树形式的汽车外造型进行特征提取,进而对汽车风阻系数进行快速预测。通过改变卷积层数与全连接层数,研究了不同卷积神经网络结构对风阻系数预测精度的影响。与参数化方法相比,本文提出的外造型表示方法能更好地描述模型细节,构建的卷积神经网络结构对风阻系数预测的最小相对误差为1.453%,且计算速度是CFD仿真的1620倍,具有较高的精度及计算效率。

多任务联合学习的图卷积神经网络推荐

基于图神经网络的协同过滤推荐可以更有效地挖掘用户项目之间的交互信息,但其性能依然受到数据稀疏和表征学习质量不高问题的影响,因此提出一种多任务联合学习的图卷积神经网络推荐(multi-task joint learning for graph convolutional neural network recommendations,MTJL-GCN)模型。利用图神经网络在用户-项目交互图上所聚集到的同质结构信息与初始嵌入信息形成结构邻居关系,设计节点邻居关系的对比学习辅助任务来缓解数据稀疏问题;向节点的原始表征添加随机的统一噪声进行表征级数据增强,构建节点表征关系的对比学习辅助任务,并提出直接优化对齐性和均匀性两个属性的学习目标来提高表征学习质量;将图协同过滤推荐任务与对比学习辅助任务和直接优化学习目标进行联合训练,从而提升推荐性能。在Amazon-books和Yelp2018两个公开数据集上进行实验,该模型在Recall@k和NDCG@k两个推荐性能指标上的表现均优于基线模型,证明了MTJL-GCN模型的有效性。

AR-MED共振特征增强的风电齿轮箱故障诊断

针对风电齿轮箱故障时脉冲成分往往淹没在其他频率分量中,早期故障特征难以有效提取的问题,提出一种自回归最小熵解卷积(AR-MED)共振特征增强的风电齿轮箱故障诊断方法,并结合一维卷积神经网络(1DCNN),实现齿轮箱高精度故障诊断。首先,使用共振稀疏分解算法(RSSD)将振动信号分解成含有噪声和谐波成分的高共振分量和含有故障冲击成分的低共振分量;其次,对低共振分量使用自回归最小熵解卷积运算,增强低共振分量中微弱的周期性冲击成分;最后,构建自回归最小熵解卷积共振特征增强的1DCNN模型,将分解得到的谐波分量和周期性冲击分量进行特征融合以及有针对的训练和分类。实验结果表明,与现有故障诊断模型相比,所提方法在提取风电齿轮箱的故障特征信息以及提高故障诊断精度方面具有有效性和优越性。