GeoNER:Geological Named Entity Recognition with Enriched Domain Pre-Training Model and Adversarial Training

As important geological data,a geological report contains rich expert and geological knowledge,but the challenge facing current research into geological knowledge extraction and mining is how to render accurate understanding of geological reports guided by domain knowledge.While generic named entity recognition models/tools can be utilized for the processing of geoscience reports/documents,their effectiveness is hampered by a dearth of domain-specific knowledge,which in turn leads to a pronounced decline in recognition accuracy.This study summarizes six types of typical geological entities,with reference to the ontological system of geological domains and builds a high quality corpus for the task of geological named entity recognition(GNER).In addition,Geo Wo BERT-adv BGP(Geological Word-base BERTadversarial training Bi-directional Long Short-Term Memory Global Pointer)is proposed to address the issues of ambiguity,diversity and nested entities for the geological entities.The model first uses the fine-tuned word granularitybased pre-training model Geo Wo BERT(Geological Word-base BERT)and combines the text features that are extracted using the Bi LSTM(Bi-directional Long Short-Term Memory),followed by an adversarial training algorithm to improve the robustness of the model and enhance its resistance to interference,the decoding finally being performed using a global association pointer algorithm.The experimental results show that the proposed model for the constructed dataset achieves high performance and is capable of mining the rich geological information.

Boosting Adversarial Training with Learnable Distribution

In recent years,various adversarial defense methods have been proposed to improve the robustness of deep neural networks.Adversarial training is one of the most potent methods to defend against adversarial attacks.However,the difference in the feature space between natural and adversarial examples hinders the accuracy and robustness of the model in adversarial training.This paper proposes a learnable distribution adversarial training method,aiming to construct the same distribution for training data utilizing the Gaussian mixture model.The distribution centroid is built to classify samples and constrain the distribution of the sample features.The natural and adversarial examples are pushed to the same distribution centroid to improve the accuracy and robustness of the model.The proposed method generates adversarial examples to close the distribution gap between the natural and adversarial examples through an attack algorithm explicitly designed for adversarial training.This algorithm gradually increases the accuracy and robustness of the model by scaling perturbation.Finally,the proposed method outputs the predicted labels and the distance between the sample and the distribution centroid.The distribution characteristics of the samples can be utilized to detect adversarial cases that can potentially evade the model defense.The effectiveness of the proposed method is demonstrated through comprehensive experiments.

A Study of Ensemble Feature Selection and Adversarial Training for Malicious User Detection

The continuously booming of information technology has shed light on developing a variety of communication networks,multimedia,social networks and Internet of Things applications.However,users inevitably suffer from the intrusion of malicious users.Some studies focus on static characteristics of malicious users,which is easy to be bypassed by camouflaged malicious users.In this paper,we present a malicious user detection method based on ensemble feature selection and adversarial training.Firstly,the feature selection alleviates the dimension disaster problem and achieves more accurate classification performance.Secondly,we embed features into the multidimensional space and aggregate it into a feature map to encode the explicit content preference and implicit interaction preference.Thirdly,we use an effective ensemble learning which could avoid over-fitting and has good noise resistance.Finally,we propose a datadriven neural network detection model with the regularization technique adversarial training to deeply analyze the characteristics.It simplifies the parameters,obtaining more robust interaction features and pattern features.We demonstrate the effectiveness of our approach with numerical simulation results for malicious user detection,where the robustness issues are notable concerns.

Instance Reweighting Adversarial Training Based on Confused Label

Reweighting adversarial examples during training plays an essential role in improving the robustness of neural networks,which lies in the fact that examples closer to the decision boundaries are much more vulnerable to being attacked and should be given larger weights.The probability margin(PM)method is a promising approach to continuously and path-independently mea-suring such closeness between the example and decision boundary.However,the performance of PM is limited due to the fact that PM fails to effectively distinguish the examples having only one misclassified category and the ones with multiple misclassified categories,where the latter is closer to multi-classification decision boundaries and is supported to be more critical in our observation.To tackle this problem,this paper proposed an improved PM criterion,called confused-label-based PM(CL-PM),to measure the closeness mentioned above and reweight adversarial examples during training.Specifi-cally,a confused label(CL)is defined as the label whose prediction probability is greater than that of the ground truth label given a specific adversarial example.Instead of considering the discrepancy between the probability of the true label and the probability of the most misclassified label as the PM method does,we evaluate the closeness by accumulating the probability differences of all the CLs and ground truth label.CL-PM shares a negative correlation with data vulnerability:data with larger/smaller CL-PM is safer/riskier and should have a smaller/larger weight.Experiments demonstrated that CL-PM is more reliable in indicating the closeness regarding multiple misclassified categories,and reweighting adversarial training based on CL-PM outperformed state-of-the-art counterparts.

Adversarial Training Against Adversarial Attacks for Machine Learning-Based Intrusion Detection Systems

Intrusion detection system plays an important role in defending networks from security breaches.End-to-end machine learning-based intrusion detection systems are being used to achieve high detection accuracy.However,in case of adversarial attacks,that cause misclassification by introducing imperceptible perturbation on input samples,performance of machine learning-based intrusion detection systems is greatly affected.Though such problems have widely been discussed in image processing domain,very few studies have investigated network intrusion detection systems and proposed corresponding defence.In this paper,we attempt to fill this gap by using adversarial attacks on standard intrusion detection datasets and then using adversarial samples to train various machine learning algorithms(adversarial training)to test their defence performance.This is achieved by first creating adversarial sample based on Jacobian-based Saliency Map Attack(JSMA)and Fast Gradient Sign Attack(FGSM)using NSLKDD,UNSW-NB15 and CICIDS17 datasets.The study then trains and tests JSMA and FGSM based adversarial examples in seen(where model has been trained on adversarial samples)and unseen(where model is unaware of adversarial packets)attacks.The experiments includes multiple machine learning classifiers to evaluate their performance against adversarial attacks.The performance parameters include Accuracy,F1-Score and Area under the receiver operating characteristic curve(AUC)Score.

Towards sustainable adversarial training with successive perturbation generation

Adversarial training with online-generated adversarial examples has achieved promising performance in defending adversarial attacks and improving robustness of convolutional neural network models.However,most existing adversarial training methods are dedicated to finding strong adversarial examples for forcing the model to learn the adversarial data distribution,which inevitably imposes a large computational overhead and results in a decrease in the generalization performance on clean data.In this paper,we show that progressively enhancing the adversarial strength of adversarial examples across training epochs can effectively improve the model robustness,and appropriate model shifting can preserve the generalization performance of models in conjunction with negligible computational cost.To this end,we propose a successive perturbation generation scheme for adversarial training(SPGAT),which progressively strengthens the adversarial examples by adding the perturbations on adversarial examples transferred from the previous epoch and shifts models across the epochs to improve the efficiency of adversarial training.The proposed SPGAT is both efficient and effective;e.g.,the computation time of our method is 900 min as against the 4100 min duration observed in the case of standard adversarial training,and the performance boost is more than 7%and 3%in terms of adversarial accuracy and clean accuracy,respectively.We extensively evaluate the SPGAT on various datasets,including small-scale MNIST,middle-scale CIFAR-10,and large-scale CIFAR-100.The experimental results show that our method is more efficient while performing favorably against state-of-the-art methods.

Identity-Preserving Adversarial Training for Robust Network Embedding

Network embedding,as an approach to learning low-dimensional representations of nodes,has been proved extremely useful in many applications,e.g.,node classification and link prediction.Unfortunately,existing network embed-ding models are vulnerable to random or adversarial perturbations,which may degrade the performance of network em-bedding when being applied to downstream tasks.To achieve robust network embedding,researchers introduce adversari-al training to regularize the embedding learning process by training on a mixture of adversarial examples and original ex-amples.However,existing methods generate adversarial examples heuristically,failing to guarantee the imperceptibility of generated adversarial examples,and thus limit the power of adversarial training.In this paper,we propose a novel method Identity-Preserving Adversarial Training(IPAT)for network embedding,which generates imperceptible adversarial exam-ples with explicit identity-preserving regularization.We formalize such identity-preserving regularization as a multi-class classification problem where each node represents a class,and we encourage each adversarial example to be discriminated as the class of its original node.Extensive experimental results on real-world datasets demonstrate that our proposed IPAT method significantly improves the robustness of network embedding models and the generalization of the learned node representations on various downstream tasks.

Mathematical Named Entity Recognition Based on Adversarial Training and Self-Attention

Mathematical named entity recognition(MNER)is one of the fundamental tasks in the analysis of mathematical texts.To solve the existing problems of the current neural network that has local instability,fuzzy entity boundary,and long-distance dependence between entities in Chinese mathematical entity recognition task,we propose a series of optimization processing methods and constructed an Adversarial Training and Bidirectional long shortterm memory-Selfattention Conditional random field(AT-BSAC)model.In our model,the mathematical text was vectorized by the word embedding technique,and small perturbations were added to the word vector to generate adversarial samples,while local features were extracted by Bi-directional Long Short-Term Memory(BiLSTM).The self-attentive mechanism was incorporated to extract more dependent features between entities.The experimental results demonstrated that the AT-BSAC model achieved a precision(P)of 93.88%,a recall(R)of 93.84%,and an F1-score of 93.74%,respectively,which is 8.73%higher than the F1-score of the previous Bi-directional Long Short-Term Memory Conditional Random Field(BiLSTM-CRF)model.The effectiveness of the proposed model in mathematical named entity recognition.

Improving the performance of SSVEP-BCI contaminated by physiological noise via adversarial training

Brain-computer interface(BCI)based on Steady-State Visual Evoked Potentials(SSVEP)provides an effective method for human-computer communication.In practical application scenarios,SSVEP-BCI systems are easily interfered by physiological noises such as electromyography(EMG)and electrooculography(EOG).The performance of traditional SSVEP recognition methods will degrade in such a noisy environment,which limits their real-world applications.To alleviate the interference of noise,existing works either require additional reference electrodes or are designed for removing background noise such as trend terms rather than physiological noises.In this study,we utilize adversarial training(AT)and neural networks(NNs)to construct a robust recognition method for SSVEP contaminated by physiological noise.During model training,we generate adversarial noises which are most harmful to the current model according to gradients and enforce the model to overcome them.In this way,we strengthen the robustness of the model to potential noises,such as physiological noises.In this study,we recorded a real-world speaking SSVEP dataset and simulated various noisy datasets to conducted comparison experiments on two benchmark models named EEGNet and DeepConvNet.The experimental results demonstrated that AT strategies can help the neural networks get better performance on SSVEP data contaminated by EMG and EOG.We also verified that introducing AT can slightly improve the performance of models under a cross-subject scenario.Our method can be integrated into existing deep learning methods efficiently and will contribute to the real-world applications of SSVEP.

LDAS&ET-AD:Learnable Distillation Attack Strategies and Evolvable Teachers Adversarial Distillation

Adversarial distillation(AD)has emerged as a potential solution to tackle the challenging optimization problem of loss with hard labels in adversarial training.However,fixed sample-agnostic and student-egocentric attack strategies are unsuitable for distillation.Additionally,the reliability of guidance from static teachers diminishes as target models become more robust.This paper proposes an AD method called Learnable Distillation Attack Strategies and Evolvable Teachers Adversarial Distillation(LDAS&ET-AD).Firstly,a learnable distillation attack strategies generating mechanism is developed to automatically generate sample-dependent attack strategies tailored for distillation.A strategy model is introduced to produce attack strategies that enable adversarial examples(AEs)to be created in areas where the target model significantly diverges from the teachers by competing with the target model in minimizing or maximizing the AD loss.Secondly,a teacher evolution strategy is introduced to enhance the reliability and effectiveness of knowledge in improving the generalization performance of the target model.By calculating the experimentally updated target model’s validation performance on both clean samples and AEs,the impact of distillation from each training sample and AE on the target model’s generalization and robustness abilities is assessed to serve as feedback to fine-tune standard and robust teachers accordingly.Experiments evaluate the performance of LDAS&ET-AD against different adversarial attacks on the CIFAR-10 and CIFAR-100 datasets.The experimental results demonstrate that the proposed method achieves a robust precision of 45.39%and 42.63%against AutoAttack(AA)on the CIFAR-10 dataset for ResNet-18 and MobileNet-V2,respectively,marking an improvement of 2.31%and 3.49%over the baseline method.In comparison to state-of-the-art adversarial defense techniques,our method surpasses Introspective Adversarial Distillation,the top-performing method in terms of robustness under AA attack for the CIFAR-10 dataset,with enhancements of 1.40%and 1.43%for ResNet-18 and MobileNet-V2,respectively.These findings demonstrate the effectiveness of our proposed method in enhancing the robustness of deep learning networks(DNNs)against prevalent adversarial attacks when compared to other competing methods.In conclusion,LDAS&ET-AD provides reliable and informative soft labels to one of the most promising defense methods,AT,alleviating the limitations of untrusted teachers and unsuitable AEs in existing AD techniques.We hope this paper promotes the development of DNNs in real-world trust-sensitive fields and helps ensure a more secure and dependable future for artificial intelligence systems.

Adversarial Training for Supervised Relation Extraction

Most supervised methods for relation extraction(RE) involve time-consuming human annotation. Distant supervision for RE is an efficient method to obtain large corpora that contains thousands of instances and various relations. However, the existing approaches rely heavily on knowledge bases(e.g., Freebase), thereby introducing data noise. Various relations and noisy labeling instances make the issue difficult to solve. In this study, we propose a model based on a piecewise convolution neural network with adversarial training. Inspired by generative adversarial networks, we adopt a heuristic algorithm to identify noisy datasets and apply adversarial training to RE. Experiments on the extended dataset of SemEval-2010 Task 8 show that our model can obtain more accurate training data for RE and significantly outperforms several competitive baseline models. Our model has an F1 score of 89.61%.

Remaining Useful Life Prediction With Partial Sensor Malfunctions Using Deep Adversarial Networks

In recent years,intelligent data-driven prognostic methods have been successfully developed,and good machinery health assessment performance has been achieved through explorations of data from multiple sensors.However,existing datafusion prognostic approaches generally rely on the data availability of all sensors,and are vulnerable to potential sensor malfunctions,which are likely to occur in real industries especially for machines in harsh operating environments.In this paper,a deep learning-based remaining useful life(RUL)prediction method is proposed to address the sensor malfunction problem.A global feature extraction scheme is adopted to fully exploit information of different sensors.Adversarial learning is further introduced to extract generalized sensor-invariant features.Through explorations of both global and shared features,promising and robust RUL prediction performance can be achieved by the proposed method in the testing scenarios with sensor malfunctions.The experimental results suggest the proposed approach is well suited for real industrial applications.

Domain adversarial training for classification of cracking in images of concrete surfaces

The development of automatic methods to recognize cracks in surfaces of concrete has been under focus in recent years,firstly through computer vision methods and more recently focusing on convolutional neural networks that are delivering promising results.Challenges are still persisting in crack recognition,namely due to the confusion added by the myriad of elements commonly found on concrete surfaces.The robustness of these methods would deal with these elements if access to correspondingly heterogeneous datasets was possible.Even so,this would be a cumbersome methodology,since training would be needed for each particular case and models would be case dependent.Thus,efforts from the scientific community are focusing on generalizing neural network models to achieve high per-formance in images from different domains,slightly different from those in which they were effectively trained.The generalization of networks can be achieved by domain adaptation techniques at the training stage.Domain adapta-tion enables finding a feature space in which features from both domains are invariant,and thus,classes become separable.The work presented here proposes the DA-Crack method,which is a domain adversarial training method,to generalize a neural network for recognizing cracks in images of concrete surfaces.The domain adversarial method uses a convolutional extractor followed by a classifier and a discriminator,and relies on two datasets:a source labeled dataset and a target unlabeled small dataset.The classifier is responsible for the classification of images randomly chosen,while the discriminator is dedicated to uncovering to which dataset each image belongs.Backpropagation from the discriminator reverses the gradient used to update the extractor.This enables fighting the convergence promoted by the updating backpropagated from the classifier,and thus generalizing the extractor enabling it for crack recognition of images from both source and target datasets.Results show that the DA-Crack training method improved accuracy in crack classification of images from the target dataset in 54 percentage points,while accuracy on the source dataset remains unaffected.

Unsupervised Anomaly Detection Approach Based on Adversarial Memory Autoencoders for Multivariate Time Series

The widespread usage of Cyber Physical Systems(CPSs)generates a vast volume of time series data,and precisely determining anomalies in the data is critical for practical production.Autoencoder is the mainstream method for time series anomaly detection,and the anomaly is judged by reconstruction error.However,due to the strong generalization ability of neural networks,some abnormal samples close to normal samples may be judged as normal,which fails to detect the abnormality.In addition,the dataset rarely provides sufficient anomaly labels.This research proposes an unsupervised anomaly detection approach based on adversarial memory autoencoders for multivariate time series to solve the above problem.Firstly,an encoder encodes the input data into low-dimensional space to acquire a feature vector.Then,a memory module is used to learn the feature vector’s prototype patterns and update the feature vectors.The updating process allows partial forgetting of information to prevent model overgeneralization.After that,two decoders reconstruct the input data.Finally,this research uses the Peak Over Threshold(POT)method to calculate the threshold to determine anomalous samples from normal samples.This research uses a two-stage adversarial training strategy during model training to enlarge the gap between the reconstruction error of normal and abnormal samples.The proposed method achieves significant anomaly detection results on synthetic and real datasets from power systems,water treatment plants,and computer clusters.The F1 score reached an average of 0.9196 on the five datasets,which is 0.0769 higher than the best baseline method.

Black Box Adversarial Defense Based on Image Denoising and Pix2Pix

Deep Neural Networks (DNN) are widely utilized due to their outstanding performance, but the susceptibility to adversarial attacks poses significant security risks, making adversarial defense research crucial in the field of AI security. Currently, robustness defense techniques for models often rely on adversarial training, a method that tends to only defend against specific types of attacks and lacks strong generalization. In response to this challenge, this paper proposes a black-box defense method based on Image Denoising and Pix2Pix (IDP) technology. This method does not require prior knowledge of the specific attack type and eliminates the need for cumbersome adversarial training. When making predictions on unknown samples, the IDP method first undergoes denoising processing, followed by inputting the processed image into a trained Pix2Pix model for image transformation. Finally, the image generated by Pix2Pix is input into the classification model for prediction. This versatile defense approach demonstrates excellent defensive performance against common attack methods such as FGSM, I-FGSM, DeepFool, and UPSET, showcasing high flexibility and transferability. In summary, the IDP method introduces new perspectives and possibilities for adversarial sample defense, alleviating the limitations of traditional adversarial training methods and enhancing the overall robustness of models.

Bearing fault diagnosis based on a multiple-constraint modal-invariant graph convolutional fusion network

Multisensor data fusionmethod can improve the accuracy of bearing fault diagnosis,in order to address the problems of single-sensor data types and the insufficient exploration of redundancy and complementarity between different modal data in most existing multisensor data fusion methods for bearing fault diagnosis,a bearing fault diagnosis method based on a Multiple-Constraint Modal-Invariant Graph Convolutional Fusion Network(MCMI-GCFN)is proposed in this paper.Firstly,a Convolutional Autoencoder(CAE)and Squeeze-and-Excitation Block(SE block)are used to extract features of raw current and vibration signals.Secondly,the model introduces source domain classifiers and domain discriminators to capture modal invariance between different modal data based on domain adversarial training,making use of the redundancy and complementarity between multimodal data.Then,the spatial aggregation property of Graph Convolutional Neural Networks(GCN)is utilized to capture the dependency relationship between current and vibration modes with similar time step features for accurately fusing contextual semantic information.Finally,the validation is conducted on the public bearing damage current and vibration dataset from Paderborn University.The experimental results showed that the delivered fusion method achieved a bearing fault diagnosis accuracy of 99.6%,which was about 9%–11.4%better than that with nonfusion methods.

GACNet: A Generative Adversarial Capsule Network for Regional Epitaxial Traffic Flow Prediction

With continuous urbanization,cities are undergoing a sharp expansion within the regional space.Due to the high cost,the prediction of regional traffic flow is more difficult to extend to entire urban areas.To address this challenging problem,we present a new deep learning architecture for regional epitaxial traffic flow prediction called GACNet,which predicts traffic flow of surrounding areas based on inflow and outflow information in central area.The method is data-driven,and the spatial relationship of traffic flow is characterized by dynamically transforming traffic information into images through a two-dimensional matrix.We introduce adversarial training to improve performance of prediction and enhance the robustness.The generator mainly consists of two parts:abstract traffic feature extraction in the central region and traffic prediction in the extended region.In particular,the feature extraction part captures nonlinear spatial dependence using gated convolution,and replaces the maximum pooling operation with dynamic routing,finally aggregates multidimensional information in capsule form.The effectiveness of the method is evaluated using traffic flow datasets for two real traffic networks:Beijing and New York.Experiments on highly challenging datasets show that our method performs well for this task.

Adversarial Neural Collaborative Filtering with Embedding Dimension Correlations

Recently,convolutional neural networks(CNNs)have achieved excellent performance for the recommendation system by extracting deep features and building collaborative filtering models.However,CNNs have been verified susceptible to adversarial examples.This is because adversarial samples are subtle non-random disturbances,which indicates that machine learning models produce incorrect outputs.Therefore,we propose a novel model of Adversarial Neural Collaborative Filtering with Embedding Dimension Correlations,named ANCF in short,to address the adversarial problem of CNN-based recommendation system.In particular,the proposed ANCF model adopts the matrix factorization to train the adversarial personalized ranking in the prediction layer.This is because matrix factorization supposes that the linear interaction of the latent factors,which are captured between the user and the item,can describe the observable feedback,thus the proposed ANCF model can learn more complicated representation of their latent factors to improve the performance of recommendation.In addition,the ANCF model utilizes the outer product instead of the inner product or concatenation to learn explicitly pairwise embedding dimensional correlations and obtain the interaction map from which CNNs can utilize its strengths to learn high-order correlations.As a result,the proposed ANCF model can improve the robustness performance by the adversarial personalized ranking,and obtain more information by encoding correlations between different embedding layers.Experimental results carried out on three public datasets demonstrate that the ANCF model outperforms other existing recommendation models.

Understanding Discrepancy of Power System Dynamic Security Assessment with Unknown Faults: A Reliable Transfer Learning-based Method

This letter proposes a reliable transfer learning(RTL)method for pre-fault dynamic security assessment(DSA)in power systems to improve DSA performance in the presence of potentially related unknown faults.It takes individual discrepancies into consideration and can handle unknown faults with incomplete data.Extensive experiment results demonstrate high DSA accuracy and computational efficiency of the proposed RTL method.Theoretical analysis shows RTL can guarantee system performance.

Generating high-resolution climatological precipitation data using SinGAN

High-resolution(HR)climate data are indispensable for studying regional climate trends,disaster prediction,and urban development planning in the face of climate change.However,state-of-the-art long-term global climate simulations do not provide appropriate HR climate data.Deep learning models are often used to obtain high-resolution climate data.However,due to the fact that these models require sufficient low-resolution(LR)and HR data pairs for the training process,they cannot be applied to scenario with inadequate training data.In this paper,we explore the applicability of a single image generative adversarial network(SinGAN)in generating HR climate data.SinGAN relies on single LR input data to obtain the corresponding HR data.To improve the performance for extreme-value regions,we propose a SinGAN combined with the weighted patchGAN discriminator(WSinGAN).The proposed WSinGAN outperforms comparable models in generating HR precipitation data,and its results are close to real HR data with sharp gradients and more refined small-scale features.We also test the scalability of the pre-trained WSinGAN for unseen samples and show that although only a single LR sample is used to train WSinGAN,it can still produce reliable HR data for unseen data.