Multi-Head Attention Spatial-Temporal Graph Neural Networks for Traffic Forecasting

Accurate traffic prediction is crucial for an intelligent traffic system (ITS). However, the excessive non-linearity and complexity of the spatial-temporal correlation in traffic flow severely limit the prediction accuracy of most existing models, which simply stack temporal and spatial modules and fail to capture spatial-temporal features effectively. To improve the prediction accuracy, a multi-head attention spatial-temporal graph neural network (MSTNet) is proposed in this paper. First, the traffic data is decomposed into unique time spans that conform to positive rules, and valuable traffic node attributes are mined through an adaptive graph structure. Second, time and spatial features are captured using a multi-head attention spatial-temporal module. Finally, a multi-step prediction module is used to achieve future traffic condition prediction. Numerical experiments were conducted on an open-source dataset, and the results demonstrate that MSTNet performs well in spatial-temporal feature extraction and achieves more positive forecasting results than the baseline methods.

Model Agnostic Meta-Learning(MAML)-Based Ensemble Model for Accurate Detection of Wheat Diseases Using Vision Transformer and Graph Neural Networks

Wheat is a critical crop,extensively consumed worldwide,and its production enhancement is essential to meet escalating demand.The presence of diseases like stem rust,leaf rust,yellow rust,and tan spot significantly diminishes wheat yield,making the early and precise identification of these diseases vital for effective disease management.With advancements in deep learning algorithms,researchers have proposed many methods for the automated detection of disease pathogens;however,accurately detectingmultiple disease pathogens simultaneously remains a challenge.This challenge arises due to the scarcity of RGB images for multiple diseases,class imbalance in existing public datasets,and the difficulty in extracting features that discriminate between multiple classes of disease pathogens.In this research,a novel method is proposed based on Transfer Generative Adversarial Networks for augmenting existing data,thereby overcoming the problems of class imbalance and data scarcity.This study proposes a customized architecture of Vision Transformers(ViT),where the feature vector is obtained by concatenating features extracted from the custom ViT and Graph Neural Networks.This paper also proposes a Model AgnosticMeta Learning(MAML)based ensemble classifier for accurate classification.The proposedmodel,validated on public datasets for wheat disease pathogen classification,achieved a test accuracy of 99.20%and an F1-score of 97.95%.Compared with existing state-of-the-art methods,this proposed model outperforms in terms of accuracy,F1-score,and the number of disease pathogens detection.In future,more diseases can be included for detection along with some other modalities like pests and weed.

Identification of Anomaly Scenes in Videos Using Graph Neural Networks

Generally,conventional methods for anomaly detection rely on clustering,proximity,or classification.With themassive growth in surveillance videos,outliers or anomalies find ingenious ways to obscure themselves in the network and make conventional techniques inefficient.This research explores the structure of Graph neural networks(GNNs)that generalize deep learning frameworks to graph-structured data.Every node in the graph structure is labeled and anomalies,represented by unlabeled nodes,are predicted by performing random walks on the node-based graph structures.Due to their strong learning abilities,GNNs gained popularity in various domains such as natural language processing,social network analytics and healthcare.Anomaly detection is a challenging task in computer vision but the proposed algorithm using GNNs efficiently performs the identification of anomalies.The Graph-based deep learning networks are designed to predict unknown objects and outliers.In our case,they detect unusual objects in the form of malicious nodes.The edges between nodes represent a relationship of nodes among each other.In case of anomaly,such as the bike rider in Pedestrians data,the rider node has a negative value for the edge and it is identified as an anomaly.The encoding and decoding layers are crucial for determining how statistical measurements affect anomaly identification and for correcting the graph path to the best possible outcome.Results show that the proposed framework is a step ahead of the traditional approaches in detecting unusual activities,which shows a huge potential in automatically monitoring surveillance videos.Performing autonomous monitoring of CCTV,crime control and damage or destruction by a group of people or crowd can be identified and alarms may be triggered in unusual activities in streets or public places.The suggested GNN model improves accuracy by 4%for the Pedestrian 2 dataset and 12%for the Pedestrian 1 dataset compared to a few state-of the-art techniques.

Collision-aware interactive simulation using graph neural networks

Deep simulations have gained widespread attention owing to their excellent acceleration performances.However,these methods cannot provide effective collision detection and response strategies.We propose a deep interac-tive physical simulation framework that can effectively address tool-object collisions.The framework can predict the dynamic information by considering the collision state.In particular,the graph neural network is chosen as the base model,and a collision-aware recursive regression module is introduced to update the network parameters recursively using interpenetration distances calculated from the vertex-face and edge-edge tests.Additionally,a novel self-supervised collision term is introduced to provide a more compact collision response.This study extensively evaluates the proposed method and shows that it effectively reduces interpenetration artifacts while ensuring high simulation efficiency.

Accurate and efficient floor localization with scalable spiking graph neural networks

Floor localization is crucial for various applications such as emergency response and rescue,indoor positioning,and recommender systems.The existing floor localization systems have many drawbacks,like low accuracy,poor scalability,and high computational costs.In this paper,we first frame the problem of floor localization as one of learning node embeddings to predict the floor label of a subgraph.Then,we introduce FloorLocator,a deep learning-based method for floor localization that integrates efficient spiking neural networks with powerful graph neural networks.This approach offers high accuracy,easy scalability to new buildings,and computational efficiency.Experimental results on using several public datasets demonstrate that FloorLocator outperforms state-of-the-art methods.Notably,in building B0,FloorLocator achieved recognition accuracy of 95.9%,exceeding state-of-the-art methods by at least 10%.In building B1,it reached an accuracy of 82.1%,surpassing the latest methods by at least 4%.These results indicate FloorLocator’s superiority in multi-floor building environment localization.

GraphFlow+:Exploiting Conversation Flow in Conversational Machine Comprehension with Graph Neural Networks

The conversation machine comprehension(MC)task aims to answer questions in the multi-turn conversation for a single passage.However,recent approaches don’t exploit information from historical conversations effectively,which results in some references and ellipsis in the current question cannot be recognized.In addition,these methods do not consider the rich semantic relationships between words when reasoning about the passage text.In this paper,we propose a novel model GraphFlow+,which constructs a context graph for each conversation turn and uses a unique recurrent graph neural network(GNN)to model the temporal dependencies between the context graphs of each turn.Specifically,we exploit three different ways to construct text graphs,including the dynamic graph,static graph,and hybrid graph that combines the two.Our experiments on CoQA,QuAC and DoQA show that the GraphFlow+model can outperform the state-of-the-art approaches.

Corporate Credit Ratings Based on Hierarchical Heterogeneous Graph Neural Networks

order to help investors understand the credit status of target corporations and reduce investment risks,the corporate credit rating model has become an important evaluation tool in the financial market.These models are based on statistical learning,machine learning and deep learning especially graph neural networks(GNNs).However,we found that only few models take the hierarchy,heterogeneity or unlabeled data into account in the actual corporate credit rating process.Therefore,we propose a novel framework named hierarchical heterogeneous graph neural networks(HHGNN),which can fully model the hierarchy of corporate features and the heterogeneity of relationships between corporations.In addition,we design an adversarial learning block to make full use of the rich unlabeled samples in the financial data.Extensive experiments conducted on the public-listed corporate rating dataset prove that HHGNN achieves SOTA compared to the baseline methods.

Towards adaptive graph neural networks via solving prior-data conflicts

Graph neural networks(GNNs)have achieved remarkable performance in a variety of graph-related tasks.Recent evidence in the GNN community shows that such good performance can be attributed to the homophily prior;i.e.,connected nodes tend to have similar features and labels.However,in heterophilic settings where the features of connected nodes may vary significantly,GNN models exhibit notable performance deterioration.In this work,we formulate this problem as prior-data confict and propose a model called the mixture-prior graph neural network(MPGNN).First,to address the mismatch of homophily prior on heterophilic graphs,we introduce the non-informative prior,which makes no assumptions about the relationship between connected nodes and learns such relationship from the data.Second,to avoid performance degradation on homophilic graphs,we implement a soft switch to balance the effects of homophily prior and non-informative prior by learnable weights.We evaluate the performance of MPGNN on both synthetic and real-world graphs.Results show that MPGNN can effectively capture the relationship between connected nodes,while the soft switch helps select a suitable prior according to the graph characteristics.With these two designs,MPGNN outperforms state-of-the-art methods on heterophilic graphs without sacrificing performance on homophilic graphs.

Investigating the Logical Capability of Graph Neural Networks via the Connection to C_(2)

Graph neural networks(GNNs)have garnered substantial application across a spectrum of real-world scenarios due to their remarkable ability to handle data organized in the form of graphs.Nonetheless,the full extent of GNNs'computational properties and logical capability remains a subject of ongoing investigation.This study undertakes an exploration of the logical capabilities intrinsic to GNNs,approaching the matter from a theoretical standpoint.In this pursuit,a pivotal connection is established between GNNs and a specific fragment of first-order logic known as C_(2),which serves as a logical framework for modeling graph data.Recent research further amplifies this discourse,introducing a subcategory of GNNs named ACR-GNN,illustrating that GNNs are capable of emulating the evaluation process of unary C,formulas.Expanding on these insights,we introduce an innovative version of GNN architectures capable of dealing with general C,formulas.To attain this,we employ a mechanism known as message passing for GNN reconstruction.The proposed GNN adaptations allow for simultaneous updating of node and node pair features,thereby enabling the management of both unary and binary C,formulas.We prove that the proposed models exhibit the equivalent expressiveness to C_(2).This underpins the profound alignment between the logical capability of GNNs and the inherent nature of the logical language C,.We conduct several experiments on both of synthetic and real-world datasets to support our claims.Through the experiments,we verify that our suggested models outperform both ACR-GNN and a commonly used model,GIN,when it comes to evaluating C,formulas.

A review on the applications of graph neural networks in materials science at the atomic scale

In recent years,interdisciplinary research has become increasingly popular within the scientific community.The fields of materials science and chemistry have also gradually begun to apply the machine learning technology developed by scientists from computer science.Graph neural networks(GNNs)are new machine learning models with powerful feature extraction,relationship inference,and compositional generalization capabilities.These advantages drive researchers to design computational models to accelerate material property prediction and new materials design,dramatically reducing the cost of traditional experimental methods.This review focuses on the principles and applications of the GNNs.The basic concepts and advantages of the GNNs are first introduced and compared to the traditional machine learning and neural networks.Then,the principles and highlights of seven classic GNN models,namely crystal graph convolutional neural networks,iCGCNN,Orbital Graph Convolutional Neural Network,MatErials Graph Network,Global Attention mechanism with Graph Neural Network,Atomistic Line Graph Neural Network,and BonDNet are discussed.Their connections and differences are also summarized.Finally,insights and prospects are provided for the rapid development of GNNs in materials science at the atomic scale.

Graph neural network-based scheduling for multi-UAV-enabled communications in D2D networks

In this paper,we jointly design the power control and position dispatch for Multi-Unmanned Aerial Vehicle(UAV)-enabled communication in Device-to-Device(D2D)networks.Our objective is to maximize the total transmission rate of Downlink Users(DUs).Meanwhile,the Quality of Service(QoS)of all D2D users must be satisfied.We comprehensively considered the interference among D2D communications and downlink transmissions.The original problem is strongly non-convex,which requires high computational complexity for traditional optimization methods.And to make matters worse,the results are not necessarily globally optimal.In this paper,we propose a novel Graph Neural Networks(GNN)based approach that can map the considered system into a specific graph structure and achieve the optimal solution in a low complexity manner.Particularly,we first construct a GNN-based model for the proposed network,in which the transmission links and interference links are formulated as vertexes and edges,respectively.Then,by taking the channel state information and the coordinates of ground users as the inputs,as well as the location of UAVs and the transmission power of all transmitters as outputs,we obtain the mapping from inputs to outputs through training the parameters of GNN.Simulation results verified that the way to maximize the total transmission rate of DUs can be extracted effectively via the training on samples.Moreover,it also shows that the performance of proposed GNN-based method is better than that of traditional means.

Adversarial attacks against dynamic graph neural networks via node injection

Dynamic graph neural networks(DGNNs)have demonstrated their extraordinary value in many practical applications.Nevertheless,the vulnerability of DNNs is a serious hidden danger as a small disturbance added to the model can markedly reduce its performance.At the same time,current adversarial attack schemes are implemented on static graphs,and the variability of attack models prevents these schemes from transferring to dynamic graphs.In this paper,we use the diffused attack of node injection to attack the DGNNs,and first propose the node injection attack based on structural fragility against DGNNs,named Structural Fragility-based Dynamic Graph Node Injection Attack(SFIA).SFIA firstly determines the target time based on the period weight.Then,it introduces a structural fragile edge selection strategy to establish the target nodes set and link them with the malicious node using serial inject.Finally,an optimization function is designed to generate adversarial features for malicious nodes.Experiments on datasets from four different fields show that SFIA is significantly superior to many comparative approaches.When the graph is injected with 1%of the original total number of nodes through SFIA,the link prediction Recall and MRR of the target DGNN link decrease by 17.4%and 14.3%respectively,and the accuracy of node classification decreases by 8.7%.

Bond sensitive graph neural networks for predicting high temperature superconductors

Finding high temperature superconductors(HTS)has been a continuing challenge due to the difficulty in predicting the transition temperature(Tc)of superconduc-tors.Recently,the efficiency of predicting Tc has been greatly improved via ma-chine learning(ML).Unfortunately,prevailing ML models have not shown adequate generalization ability to find new HTS,yet.In this work,a graph neural network model is trained to predict the maximal Tc(Tc max)of various materials.Our model reveals a close connection between Tc max and chemical bonds.It sug-gests that shorter bond lengths are favored by high Tc,which is in coherence with previous domain knowledge.More importantly,it also indicates that chemical bonds consisting of some specific chemical elements are responsible for high Tc,which is new even to the human experts.It can provide a convenient guidance to the materials scientists in search of HTS.

An End-To-End Hyperbolic Deep Graph Convolutional Neural Network Framework

Graph Convolutional Neural Networks(GCNs)have been widely used in various fields due to their powerful capabilities in processing graph-structured data.However,GCNs encounter significant challenges when applied to scale-free graphs with power-law distributions,resulting in substantial distortions.Moreover,most of the existing GCN models are shallow structures,which restricts their ability to capture dependencies among distant nodes and more refined high-order node features in scale-free graphs with hierarchical structures.To more broadly and precisely apply GCNs to real-world graphs exhibiting scale-free or hierarchical structures and utilize multi-level aggregation of GCNs for capturing high-level information in local representations,we propose the Hyperbolic Deep Graph Convolutional Neural Network(HDGCNN),an end-to-end deep graph representation learning framework that can map scale-free graphs from Euclidean space to hyperbolic space.In HDGCNN,we define the fundamental operations of deep graph convolutional neural networks in hyperbolic space.Additionally,we introduce a hyperbolic feature transformation method based on identity mapping and a dense connection scheme based on a novel non-local message passing framework.In addition,we present a neighborhood aggregation method that combines initial structural featureswith hyperbolic attention coefficients.Through the above methods,HDGCNN effectively leverages both the structural features and node features of graph data,enabling enhanced exploration of non-local structural features and more refined node features in scale-free or hierarchical graphs.Experimental results demonstrate that HDGCNN achieves remarkable performance improvements over state-ofthe-art GCNs in node classification and link prediction tasks,even when utilizing low-dimensional embedding representations.Furthermore,when compared to shallow hyperbolic graph convolutional neural network models,HDGCNN exhibits notable advantages and performance enhancements.

NeurstrucEnergy:A bi-directional GNN model for energy prediction of neural networks in IoT

A significant demand rises for energy-efficient deep neural networks to support power-limited embedding devices with successful deep learning applications in IoT and edge computing fields.An accurate energy prediction approach is critical to provide measurement and lead optimization direction.However,the current energy prediction approaches lack accuracy and generalization ability due to the lack of research on the neural network structure and the excessive reliance on customized training dataset.This paper presents a novel energy prediction model,NeurstrucEnergy.NeurstrucEnergy treats neural networks as directed graphs and applies a bi-directional graph neural network training on a randomly generated dataset to extract structural features for energy prediction.NeurstrucEnergy has advantages over linear approaches because the bi-directional graph neural network collects structural features from each layer's parents and children.Experimental results show that NeurstrucEnergy establishes state-of-the-art results with mean absolute percentage error of 2.60%.We also evaluate NeurstrucEnergy in a randomly generated dataset,achieving the mean absolute percentage error of 4.83%over 10 typical convolutional neural networks in recent years and 7 efficient convolutional neural networks created by neural architecture search.Our code is available at https://github.com/NEUSoftGreenAI/NeurstrucEnergy.git.

GACP:graph neural networks with ARMA flters and a parallel CNN for hyperspectral image classification

In recent years,the use of convolutional neural networks(CNNs)and graph neural networks(GNNs)to identify hyperspectral images(HSIs)has achieved excellent results,and such methods are widely used in agricultural remote sensing,geological exploration,and marine remote sensing.Although many generalization classification algorithms are designed for the purpose of learning a small number of samples,there is often a problem of a low utilization rate of position information in the empty spectral domain.Based on this,a GNN with an autoregressive moving average(ARMA)-based smoothingfilter samples the node information in the null spectral domain and then captures the spatial information at the pixel level via spatial feature convolution;then,the null spectral domain position information lost by the CNN is located by a coordinate attention(CA)mechanism.Finally,autoregressive,spatial convolution,and CA mechanisms are combined into multiscale features to enhance the learning capacity of the network for tiny samples.Experiments conducted on the widely used Indian Pines(IP)dataset,the Botswana(BS)dataset,Houton 2013(H2013),and the WHU-Hi-HongHu(WHU)benchmark HSI dataset demonstrate that the proposed GACP technique can perform classification work with good accuracy even with a small number of training examples.

A Review of Graph Neural Networks and Their Applications in Power Systems

Deep neural networks have revolutionized many machine learning tasks in power systems,ranging from pattern recognition to signal processing.The data in these tasks are typically represented in Euclidean domains.Nevertheless,there is an increasing number of applications in power systems,where data are collected from non-Euclidean domains and represented as graph-structured data with high-dimensional features and interdependency among nodes.The complexity of graph-structured data has brought significant challenges to the existing deep neural networks defined in Euclidean domains.Recently,many publications generalizing deep neural networks for graph-structured data in power systems have emerged.In this paper,a comprehensive overview of graph neural networks(GNNs)in power systems is proposed.Specifically,several classical paradigms of GNN structures,e.g.,graph convolutional networks,are summarized.Key applications in power systems such as fault scenario application,time-series prediction,power flow calculation,and data generation are reviewed in detail.Furthermore,main issues and some research trends about the applications of GNNs in power systems are discussed.

Dynamic interwell connectivity analysis of multi-layer waterflooding reservoirs based on an improved graph neural network

The analysis of interwell connectivity plays an important role in the formulation of oilfield development plans and the description of residual oil distribution. In fact, sandstone reservoirs in China's onshore oilfields generally have the characteristics of thin and many layers, so multi-layer joint production is usually adopted. It remains a challenge to ensure the accuracy of splitting and dynamic connectivity in each layer of the injection-production wells with limited field data. The three-dimensional well pattern of multi-layer reservoir and the relationship between injection-production wells can be equivalent to a directional heterogeneous graph. In this paper, an improved graph neural network is proposed to construct an interacting process mimics the real interwell flow regularity. In detail, this method is used to split injection and production rates by combining permeability, porosity and effective thickness, and to invert the dynamic connectivity in each layer of the injection-production wells by attention mechanism.Based on the material balance and physical information, the overall connectivity from the injection wells,through the water injection layers to the production layers and the output of final production wells is established. Meanwhile, the change of well pattern caused by perforation, plugging and switching of wells at different times is achieved by updated graph structure in spatial and temporal ways. The effectiveness of the method is verified by a combination of reservoir numerical simulation examples and field example. The method corresponds to the actual situation of the reservoir, has wide adaptability and low cost, has good practical value, and provides a reference for adjusting the injection-production relationship of the reservoir and the development of the remaining oil.

Learning to Branch in Combinatorial Optimization With Graph Pointer Networks

Traditional expert-designed branching rules in branch-and-bound(B&B) are static, often failing to adapt to diverse and evolving problem instances. Crafting these rules is labor-intensive, and may not scale well with complex problems.Given the frequent need to solve varied combinatorial optimization problems, leveraging statistical learning to auto-tune B&B algorithms for specific problem classes becomes attractive. This paper proposes a graph pointer network model to learn the branch rules. Graph features, global features and historical features are designated to represent the solver state. The graph neural network processes graph features, while the pointer mechanism assimilates the global and historical features to finally determine the variable on which to branch. The model is trained to imitate the expert strong branching rule by a tailored top-k Kullback-Leibler divergence loss function. Experiments on a series of benchmark problems demonstrate that the proposed approach significantly outperforms the widely used expert-designed branching rules. It also outperforms state-of-the-art machine-learning-based branch-and-bound methods in terms of solving speed and search tree size on all the test instances. In addition, the model can generalize to unseen instances and scale to larger instances.

Physical information-enhanced graph neural network for predicting phase separation

Although phase separation is a ubiquitous phenomenon, the interactions between multiple components make it difficult to accurately model and predict. In recent years, machine learning has been widely used in physics simulations. Here,we present a physical information-enhanced graph neural network(PIENet) to simulate and predict the evolution of phase separation. The accuracy of our model in predicting particle positions is improved by 40.3% and 51.77% compared with CNN and SVM respectively. Moreover, we design an order parameter based on local density to measure the evolution of phase separation and analyze the systematic changes with different repulsion coefficients and different Schmidt numbers.The results demonstrate that our model can achieve long-term accurate predictions of order parameters without requiring complex handcrafted features. These results prove that graph neural networks can become new tools and methods for predicting the structure and properties of complex physical systems.